Methods, systems, and media for modifying the presentation of contextually relevant documents in browser windows of a browsing application

ABSTRACT

Methods, systems, and media for presenting contextually relevant information are provided. In some implementations, the method includes: receiving information associated with a user of a user device from multiple data sources, where the user device comprises a display; identifying, without user intervention, a relevant document based on the received information associated with the user of the user device; determining that a new browser window or a new browser tab has been opened by a browser application being executed by the user device; and causing, without user intervention, the relevant document to be presented using the new browser window or new browser tab.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/619,843, filed Feb. 11, 2015, which is hereby incorporated byreference herein in its entirety.

This application is related to U.S. patent application Ser. No.14/619,827, entitled “METHODS, SYSTEMS, AND MEDIA FOR AMBIENT BACKGROUNDNOISE MODIFICATION BASED ON MOOD AND/OR BEHAVIOR INFORMATION,” (AttorneyDocket No. 0715150.282-US1), U.S. patent application Ser. No.14/9619,866, entitled “METHODS, SYSTEMS, AND MEDIA FOR PRODUCING SENSORYOUTPUTS CORRELATED WITH RELEVANT INFORMATION,” (Attorney Docket No.0715150.283-US1), U.S. patent application Ser. No. 14/619,894, entitled“METHODS, SYSTEMS, AND MEDIA FOR PRESENTING INFORMATION RELATED TO ANEVENT BASED ON METADATA,” (Attorney Docket No. 0715150.284-US1), U.S.patent application Ser. No. 14/619,821, entitled “METHODS, SYSTEMS, ANDMEDIA FOR RECOMMENDING COMPUTERIZED SERVICES BASED ON AN ANIMATE OBJECTIN THE USER'S ENVIRONMENT,” (Attorney Docket No. 0715150.285-US1), andU.S. patent application Ser. No. 14/619,863, entitled “METHODS, SYSTEMS,AND MEDIA FOR PERSONALIZING COMPUTERIZED SERVICES BASED ON MOOD AND/ORBEHAVIOR INFORMATION FROM MULTIPLE DATA SOURCES,” (Attorney Docket No.0715150.287-US1), all of which were filed on Feb. 11, 2015, andincorporated by reference herein in their entireties.

TECHNICAL FIELD

The disclosed subject matter relates to methods, systems, and media forpresenting contextually relevant information.

BACKGROUND

Many millions of documents such as web pages, emails, attachments toemails, images sent to a user, etc., are available to users of devicessuch as personal computers, mobile phones and tablet computers. However,to find a particular document, the user often must sift through manydocuments that are not the document that the user is currentlyinterested in.

Accordingly, it is desirable to provide methods, systems, and media forpresenting contextually relevant information.

SUMMARY

In accordance with some implementations of the disclosed subject matter,methods, systems, and media for presenting contextually relevantinformation are provided.

In accordance with some implementations of the disclosed subject matter,a method for presenting contextually relevant information is provided,the method comprising: receiving, using a hardware processor,information associated with a user of a user device from a plurality ofdata sources, wherein the user device comprises a display; identifying,without user intervention, a relevant document based on the receivedinformation associated with the user of the user device; determiningthat a new browser window or a new browser tab has been opened by abrowser application being executed by the user device; and causing,without user intervention, the relevant document to be presented usingthe new browser window or new browser tab.

In accordance with some implementations of the disclosed subject matter,a system for presenting contextually relevant information is provided,the system comprising: a hardware processor that is programmed to:receive information associated with a user of a user device from aplurality of data sources, wherein the user device comprises a display;identify, without user intervention, a relevant document based on thereceived information associated with the user of the user device;determine that a new browser window or a new browser tab has been openedby a browser application being executed by the user device; and cause,without user intervention, the relevant document to be presented usingthe new browser window or new browser tab.

In accordance with some implementations of the disclosed subject matter,a non-transitory computer-readable medium containing computer executableinstructions that, when executed by a processor, cause the processor toperform a method for presenting contextually relevant information isprovided, the method comprising: receiving information associated with auser of a user device from a plurality of data sources, wherein the userdevice comprises a display; identifying, without user intervention, arelevant document based on the received information associated with theuser of the user device; determining that a new browser window or a newbrowser tab has been opened by a browser application being executed bythe user device; and causing, without user intervention, the relevantdocument to be presented using the new browser window or new browsertab.

In accordance with some implementations of the disclosed subject matter,a system for presenting contextually relevant information is provided,the system comprising: means for receiving information associated with auser of a user device from a plurality of data sources, wherein the userdevice comprises a display; means for identifying, without userintervention, a relevant document based on the received informationassociated with the user of the user device; means for determining thata new browser window or a new browser tab has been opened by a browserapplication being executed by the user device; and means for causing,without user intervention, the relevant document to be presented usingthe new browser window or new browser tab.

In some implementations, the relevant document is a web page.

In some implementations, the plurality of data sources includes metadataof a plurality of messages viewed using a messaging account associatedwith the user.

In some implementations, the relevant document is a document attached toat least one of the messages, and wherein the relevant document isidentified based on metadata indicating that the relevant document wasattached to the at least one of the messages.

In some implementations, the relevant document is identified based ontiming information related to how recently the relevant document wasattached to the message.

In some implementations, the relevant document is identified based on anumber of messages in a thread of messages that includes the at leastone message to which the relevant document was attached being equal toor greater than a threshold number of messages.

In some implementations, attachment to a message is indicated by a linkto the relevant document being included in the message.

BRIEF DESCRIPTION OF THE DRAWINGS

Various objects, features, and advantages of the disclosed subjectmatter can be more fully appreciated with reference to the followingdetailed description of the disclosed subject matter when considered inconnection with the following drawings, in which like reference numeralsidentify the like elements.

FIG. 1 shows an example of a generalized schematic diagram of a systemon which the mechanisms for personalizing computerized services based onmood and/or behavior information from multiple data sources inaccordance with some implementations of the disclosed subject matter.

FIG. 2 shows a more particular example of a server of FIG. 1 that canreceive various types of data from multiple data sources and that canrecommend various types of actions based on a portion of the receiveddata to various user devices associated with a user device in accordancewith some implementations of the disclosed subject matter.

FIG. 3 shows a detailed example of hardware that can be used toimplement one or more of the user devices and servers depicted in FIG. 1in accordance with some implementations of the disclosed subject matter.

FIG. 4 shows an illustrative example of a process for personalizingcomputerized services based on mood and/or behavior information frommultiple data sources in accordance with some implementations of thedisclosed subject matter.

FIG. 5 shows an illustrative example of a user interface for promptingthe user of the user device to provide an objective in accordance withsome implementations of the disclosed subject matter.

FIG. 6 shows an illustrative example of a user interface for promptingthe user of the user device to select one or more data sources forretrieving data relating to the user in accordance with someimplementations of the disclosed subject matter.

FIG. 7 shows an illustrative example of a portion of data at varioustimes selected from the data that is received from multiple data sourcesbased on a particular objective or goal in accordance with someimplementations of the disclosed subject matter.

FIG. 8 shows an illustrative example of a user interface for presentingthe user with selectable goals corresponding to an objective inaccordance with some implementations of the disclosed subject matter.

FIG. 9 shows an illustrative example of a user interface for presentingthe user with selectable output devices that can be used to execute arecommended action in accordance with some implementations of thedisclosed subject matter.

FIG. 10 shows an illustrative example of a user interface for presentingthe user with a recommendation interface that includes a recommendedaction in accordance with some implementations of the disclosed subjectmatter.

FIG. 11 shows an illustrative example of a user interface that promptsthe user to provide feedback relating to an executed action inaccordance with some implementations of the disclosed subject matter.

FIG. 12 shows an example of a process for presenting contextuallyrelevant information to a user based on information from multiple datasources in accordance with some implementations of the disclosed subjectmatter.

FIG. 13 shows an example of a process for presenting documents relevantto communications sent and/or viewed using a computing device associatedwith the user when the computing device or another computing deviceassociated with the same user is powered on, unlocked, opened, wokenfrom a sleep mode, initiated, etc., and/or when a new browser window,and/or a new browser tab are opened in accordance with someimplementations of the disclosed subject matter.

FIG. 14 shows an example of a process for presenting documents relevantto a scheduled task associated with the user at a time associated withthe task in accordance with some implementations of the disclosedsubject matter.

FIG. 15 shows an example of a process for grouping documents based oninformation from multiple data sources in accordance with someimplementations of the disclosed subject matter.

FIG. 16 shows an example of a process for presenting a relevant documenton a second display in accordance with some implementations of thedisclosed subject matter.

DETAILED DESCRIPTION

In accordance with some implementations, as described in more detailbelow, mechanisms, which can include methods, systems, and/or computerreadable media, for personalizing computerized services based on moodand/or behavior information from multiple data sources are provided.

Generally speaking, these mechanisms can receive inputs from a user of auser device relating to a particular objective for that user and, basedon received data that relates to the user, can provide personalized andcomputerized services that may assist the user in reaching thatparticular objective. For example, the user of the user device mayindicate via a user interface a desire to incorporate more exerciseduring the course of a workday and, based on data relating to the userfrom one or more data sources for which the user has affirmatively givenconsent, the mechanisms can recommend one or more actions that maytechnologically assist the user in reaching the particularobjective—e.g., by recommending through an automatic mobile alert ornotification that the user walk to the office today based on weatherinformation, scheduling constraints based on an online calendar, and/ortraffic and public transportation information received through one ormore computer networks; by recommending that the user who has aparticular interest in flowers visit a gardening vendor along the user'sroute as identified by a computer map routing service (e.g., GoogleMaps); etc.

It should be noted that, additionally or alternatively to receiving aninput relating to a particular objective, the mechanisms can receiveuser feedback and, based on the received user feedback, determine goalsfor the user. For example, the user may indicate a lack of energy onweekdays via a user interface on the user device and the mechanisms caninterpret such an indication and determine various goals for the user,such as increasing the amount of exercise-related activities. In anotherexample, the user can be provided with an interface that requests theuser provides feedback as to the user's general mood, emotional state,and/or behavioral disposition and the mechanisms can determine goals forthe user based on the provided feedback. Illustrative examples of goalsthat can be determined for a user and/or associated with a user devicecan include reducing stress from a currently indicated stress level,generally losing weight, losing ten pounds, attaining a particular moodor emotional state (e.g., relaxed, lively, etc.), increasing the amountof exercise that the user currently achieves, making more friends,and/or any other suitable goal relating to the user's general mood,emotional state, and/or behavioral disposition.

It should also be noted that, prior to analyzing data relating to theuser from multiple data sources, determining a mood, emotional state,and/or behavioral disposition associated with the user, and/orrecommending one or more actions to the user, the mechanisms can request(or require) that the user affirmatively provide consent orauthorization to perform such determinations. For example, upon loadingan application on a mobile device, the application can prompt the userto provide authorization for receiving information from one or more datasources, performing such determinations, and/or recommending one or moreactions to the user. In a more particular example, in response todownloading the application and loading the application on the mobiledevice, the application executing on the mobile device can performdevice discovery functions to determine devices that are connected to ornearby the mobile device, such as a media playback device that includesmedia data (e.g., watch history, recorded media content information,etc.) and/or a scent generator that includes an activity and/or lightsensor for obtaining information relating to the environment around theconnected mobile device. The application can then present the user withan interface that requests (or requires) that the user affirmativelyprovide consent to accessing information from these devices by selectingthe one or more devices or data sources for receiving informationrelating to the user that can be used to determine a mood, emotionalstate, and/or behavioral disposition associated with the user, determineone or more goals or objectives associated with the user, and/orrecommend one or more actions that may impact the physical state,emotional state, and/or behavioral disposition associated with the user.Additionally or alternatively, in response to installing the applicationon the mobile device, the user can be prompted with a permission messagethat requests (or requires) that the user affirmatively provide consentprior to receiving information from one or more data sources, performingsuch determinations, and/or recommending one or more actions to theuser.

Upon receiving consent and/or authorization, the mechanisms can receiveany suitable data relating to the user from multiple data sources. Suchdata can include contextual data, social data, personal data, etc. Forexample, the mechanisms can predict a current mood state for the userbased on content and/or information published by the user on a socialnetworking service (e.g., using a social networking application on theuser device), biometric data associated with the user (e.g., from awearable computing device associated with a user account), location dataassociated with the user (e.g., from the user device), and/or any othersuitable data indicative of current mood and/or behavior of the user. Inanother example, the mechanisms can determine particular activities thatthe user has engaged in, such as attending a social event (e.g., aconference, a party, a sports event, etc. from an online calendar),consuming a media content item (e.g., a video clip, a song, a newsarticle, a webpage, etc.), interacting with a computing device (e.g., amobile phone, a wearable computing device, a tablet computer, etc.),interacting with an application (e.g., a media playback application, asocial networking application, a messaging application, a web browser,etc. on a user device), and/or any other suitable activity. Thisactivity data can, for example, be used to determine reference behaviorsassociated with the user (e.g., a particular time and portion of the dayis typically spent watching videos on a media playback applicationexecuting on a mobile device).

In some implementations, based on data relating to the user that isreceived from one or more data sources for which the user hasaffirmatively provided consent, the mechanisms can recommend one or morecomputerized actions that may assist the user in reaching one or more ofthe objectives and/or goals. For example, the mechanisms can use devicediscovery functions to determine which output devices for executing theone or more recommended actions are connected to the mobile device orare proximate to the mobile device, such as devices having a speakerthat are capable of playing audio content, devices having a display thatare capable of presenting video content, lighting systems that arecapable of providing a particular lighting scheme, and scent generatorsthat are capable of emitting a particular scent. In response, thesemechanisms can transmit instructions to an output device that is capableof executing a recommended action. For example, in response todetermining information indicative of the user's general mood, emotionalstate, and/or behavioral disposition from one or more data sources, themechanisms can identify one or more activities that, if performed, maymove the user towards a particular objective or goal. In this example,tlhe mechanisms can transmit a message or other suitable interfaceindicating the recommended activities to the mobile device associatedwith the user.

In a more particular example, in response to receiving social networkingdata from a social media application that indicates the user may beexperiencing low energy levels (e.g., analyzing text from a post usingthe social media application) and online calendar data that includesscheduling information associated with a user for a given day, themechanisms can recommend one or more computerized actions to the userthat may assist the user in reaching one or more of the determinedgoals. The mechanisms can review a route to an event listed in thecalendar data, where the route has been identified by a computer maprouting service, and transmit an interface to be presented on the mobiledevice associated with the user, where the interface recommends that theuser walk to the event and visit a particular juice vendor along theroute as identified by a computer map routing service.

Additionally or alternatively to a recommendation interface thatincludes a message or other suitable content, personalized andcomputerized services can include a determination that a particularatmosphere should be created that may affect the user's general mood,emotional state, and/or behavioral disposition. In one particularexample, the atmosphere can include causing particular content to beautomatically played back (e.g., a particular song that is designated asbeing inspirational to users), causing a news feed of articles that aredesignated as positive stories to be presented, causing photographs orother image content that are designated as amusing to users to bepresented, and sound effects that are designated as having a relaxingeffect on users, to be presented to the user on an associated userdevice (e.g., mobile device, television device, etc.). In anotherparticular example, the atmosphere can be created by accessing alighting system associated with the user or user device and causingparticular light sources to switch on or off, select the level of lightemitted from particular lighting devices, and/or select the colortemperature of particular light sources, thereby modifying the lightingscheme in the user's surroundings. In yet another example, theatmosphere can be created by modifying an ambient noise emitted by adevice connected to the user device (e.g., modifying the speed of a fanon a computing device associated with the user), emitting a particularscent from a device connected to the user device (e.g., causing a devicethat is capable of emitting particular scents and that is within aparticular proximity of the user of the user device to emit a lavenderscent), controlling an appliance or a home automation device connectedto the user device (e.g., controlling the compressor of an HVAC unit ormodifying the speed of the drum of a washer), etc.

In some implementations, the mechanisms can generate one or moreprofiles associated with a user device. For example, in someimplementations, the mechanisms can generate various profiles that canbe used to determine recommended actions suitable for the user of theuser device. For example, the mechanisms can generate a profile that isindicative of the user's current mood, emotional state, and/orbehavioral disposition and compare the generated profile with a targetprofile to determine a recommended action that, if performed, may movethe user towards an objective or goal. In a more particular example, thetarget profile can be generated based on profiles or other actions ofusers that have indicated the achievement of a particular objective orgoal (e.g., users that deem themselves to be highly successful, usersthat have lost five pounds in the past 30 days, etc.). In this example,the mechanisms can determine actions that are performed by users of userdevices determined to have achieved a particular objective or goal andcan determine whether one or more of these actions can be recommended tothe user so that the user also achieves the particular objective orgoal.

These and other features are further described in connection with FIGS.1-16.

Turning to FIG. 1, FIG. 1 shows an example 100 of a generalizedschematic diagram of a system on which the mechanisms for personalizingcomputerized services based on mood and/or behavior information frommultiple data sources can be implemented in accordance with someimplementations of the disclosed subject matter. As illustrated, system100 can include one or more user devices 102. User devices 102 can belocal to each other or remote from each other. User devices 102 can beconnected by one or more communications links 104 to a communicationnetwork 106 that can, in turn, be linked to a server 120 via acommunications link 112.

System 100 can include one or more data sources 110. Data source 110 canbe any suitable device that can gather and/or provide data relating to auser or a user device.

For example, data source 110 can include any suitable sensor that cangather and/or provide data relating to a user, such as an image sensor(e.g., a camera, a video recorder, etc.), an audio sensor (e.g., amicrophone, a sound lever meter, etc.), a radio-frequency identification(RFID) sensor, a Global Positioning System (GPS), a sensor that iscapable of measuring one or more biological parameters (e.g., a heartrate, a respiration rate, a blood pressure, a body temperature, skinmoisture, etc.), a wearable pedometer, a Wi-Fi router, etc. In a moreparticular example, data source 110 can be multiple sensors connected toa home automation system registered to a user account associated withthe user, where different data streams relating to the environment inthe user's home can be received. In another more particular example,data source 110 can include multiple sensors connected to a coffee shopthat is local to the user and an application program interface thatallows the recommendation system to request data relating to the localcoffee shop (e.g., how many patrons are currently in the shop based on adoor sensor, how many patrons are currently waiting in line to order inthe shop based on an image sensor, whether a group that the user isassigned is likely to visit the shop by comparing a target profile witha customer profile associated with the shop, etc.).

In another example, data source 110 can include a computing device, suchas a desktop, a laptop, a mobile phone, a tablet computer, a wearablecomputing device, etc. Examples of data provided by such a computingdevice can include user generated data (e.g., text inputs, photos, touchinputs, etc.), user application generated data (e.g., data provided by abrowser application, data provided by a social networking application, amessaging application, a photo sharing application, a video sharingapplication, a media player application, etc.), data generated by one ormore sensors resident on the computing device (e.g., an image sensor, aGPS, a sensor that is capable of measuring one or more biologicalparameters, etc.), and/or any other suitable data relating to the user.For example, data source 110 can include a computing device that hasbeen registered to a user having a user account and data can includedata from various applications installed on the computing device andregistered using the same user account. In this example, the user of thecomputing device can select which applications or which data types(e.g., location data, wireless network data, etc.) is used by anapplication executing on user device 102 or server 120.

In yet another example, data source 110 can include one or more servicesthat can provide data related to the user and/or a group of users (whomay be similar to the user or dissimilar from the user). Such servicescan include, for example, a web search service, a social networkingservice, a messaging service, a video sharing service, a photo sharingservice, a file hosting service, etc. In such an example, user device102 or server 120 can communicate with data source 110 via one or moreapplication programming interfaces and/or any other suitable dataexchange mechanisms.

It should be noted that data from one or more data sources 110 can beused to determine the impact of a recommended action on a user'sphysical or emotional state. The emotional state of a user can be acomplex phenomenon. Emotion can be a mental state that is associatedwith physiological activity and can be influenced by internal orexternal environmental conditions. Emotion can be associated withpersonality, mood, temperament, disposition, and motivation of a user.For example, emotional states can include happiness, contentment,tranquility, surprise, anger, fear, sadness, depression, disgust, tired,anxious, hurried, etc. In some examples, emotional states can be broadlyclassified into positive and negative emotions, where positive emotionscan include happiness and contentment and negative emotions can includeanger and depression. In addition, an example of an internalenvironmental condition includes an old memory and an example ofexternal stimulus includes stress or the relief of stress due to variousenvironmental factors.

It should also be noted that the physical or emotional state of a usercan be considered an overall snapshot or view of the user's physicalcharacteristics or emotions at a point in time. Because multiple factorscan be involved in a user's physical or emotional state, the physical oremotional state can fluctuate even over short periods of time. By usingdata relating to the user from multiple sources, a user's physical oremotional state can be predicted, which can be used to determine whetherto recommend a particular action at a given time. Moreover, changes to auser's physical or emotional state can be predicted based on new orupdated data relating to the user from multiple sources. Even further,changes to a user's physical or emotional state can be used to evaluatewhether recommended actions to devices possessed by, or locatedproximate to, the user may be moving the user towards a goal orobjective.

Data sources 110 can be local to each other or remote from each other.Each data source 110 can be connected by one or more communicationslinks 108 to communication network 106 that can, in turn, be linked toserver 120 via communications link 112 and/or user device 102 viacommunications link 104.

It should be noted that, in some implementations, prior to accessinginformation from various data sources 110, user device 102 can request(or require) that the user of user device 102 provide authorization toaccess each of the various data sources 110. In some implementations,user device 102 can detect data sources 110 that are available toprovide data relating to the user and can provide a user interface thatallows the user of user device 102 to select which data sources 110 areto be used for obtaining data relating to the user.

FIG. 2 shows an illustrative example of types of input data that can bereceived by user device 102 and/or server 120. As shown in FIG. 2,server 120 can include a data storage engine 122 for requesting,consolidating, storing, and/or processing data relating to a user or agroup of users and a data processing engine 124 for categorizingreceived data (e.g., contextual data, social data, general data, etc.),selecting particular portions of data that may be indicative of aphysical or emotional state of the user, and processing the selectedportions of data. For example, as also shown in FIG. 2, server 120 canreceive, among other things, various types of video data, text data,RFID data, ambient audio data (or keywords extracted from ambient audiodata), and mobile device data.

Using one or more data sources 110, data storage engine 122 can receiveany suitable data. For example, from one or more data sources 110, datastorage engine 112 can receive and/or request data relating toactivities engaged in by one or more users, such as “took a walk” andthe distance traversed, visited a location that corresponds with acoffee shop, attended a social event (e.g., a conference, a party, asporting event, etc.), attended a fitness training session, etc. Asanother example, from one or more data sources 110, data storage engine112 can receive and/or request data that includes timing informationrelated to an activity, such as a duration of the activity, a timecorresponding to the activity, etc. As yet another example, data storageengine 112 can receive and/or request data that includes a number ofoccurrences of an activity engaged in by one or more users during agiven time period (e.g., a day of the week, a couple of days, weekdays,weekends, etc.), a number of users that engage in a given activity,and/or any other suitable information relating to frequency informationrelated to a given activity.

Using one or more data sources 110 that include social data sources,data storage engine 122 can receive and/or request data relating tocontent and/or information published by the user on a social networkingservice. For example, the data can include one or more mood statespublished by a user on a service (e.g., a social networking service, amessaging service, a video sharing service, a photo sharing service, anelectronic commerce service, etc.). As another example, the data caninclude comments, messages, posts, locations, and/or any other suitablecontent published by the user on a social networking service. As stillanother example, the data can include any suitable information relatingto one or more social connections of the user on a social networkingservice, content posted by the social connections, locations associatedwith the social connections, etc.

Using one or more data sources 110, data storage engine 112 can receiveand/or request data relating to user interactions with one or more mediacontent items. For example, the data can include any suitableinformation relating to a media content item with which the user hasinteracted. In a more particular example, the data can include a type ofthe media content item, a description of the media content item, a linkto the media content item (e.g., a URL), an identifier that can identifythe media content item (e.g., a URI, a program identifier, etc.), anauthor of the media content item, an artist related to the media contentitem, etc. As another example, the data can include any suitableinformation about a type of a user interaction with a media contentitem, such as consuming the media content item, publishing the mediacontent item via a social networking service or any other suitableservice, sharing the media content item with other users, liking themedia content item via a social networking service or any other suitableservice, commenting on the media content item, etc. As yet anotherexample, the data can include any suitable timing information related toa user interaction with a media content item, such as a duration of theuser interaction, a time corresponding to the user interaction, etc.

Using one or more data sources 110, data storage engine 112 can receiveand/or request biometric data associated with the user. For example, inresponse to receiving authorization to access biometric data from datasource 110 that includes a sensor, the biometric data can include anysuitable physiological parameter associated with the user, such as aheart rate, a respiration rate, a blood pressure, a body temperature,skin moisture, etc. As another example, the biometric data can include arange of physiological parameters, such as a heart rate range, a bloodpressure range, etc.

Using one or more data sources 110, data storage engine 112 can receiveand/or request location data associated with the user. For example, inresponse to receiving authorization to access location information, thelocation data can include any suitable information that can be used toestimate a location of a computing device associated with the user, suchas an identifier associated with the computing device (e.g., an IPaddress, GPS signals generated by the computing device, Wi-Fi accesspoints associated with the computing device, information about a celltower to which the computing device is connected, etc. As anotherexample, the location data can include any suitable information that canbe used to estimate a location of the user, such as a location publishedby the user using a suitable service (e.g., a social networkingservice), a location that a user intends to visit (e.g., a locationassociated with a social event scheduled using a calendar applicationexecuting on a mobile device, a social network account associated withthe user, etc.), etc.

In some implementations, data storage engine 112 can categorize and/orclassify data received from data sources 110.

For example, data storage engine 122 can receive data from multiple datasources 110 (e.g., using one or more application programming interfaces)and data processing engine 124 can classify the received data as generaldata when the received data includes information about one or moreservices used by a user (e.g., a social networking service, an emailservice, a messaging service, a video sharing service, etc.), searchhistory associated with a user (e.g., keywords inputted by the user),etc.

As another example, data storage engine 122 can receive data frommultiple data sources 110 (e.g., using one or more applicationprogramming interfaces) and data processing engine 124 can classify thereceived data as contextual data when the received data includesinformation about a location of user device 102, traffic information,weather information based on location information from user device 102(e.g., “sunny,” “cold,” etc.), population density information within agiven location, a location context relating to data provided by a datasource 110 (e.g., “work,” “home,” “vacation,” etc.), and/or any othersuitable information that can provide contextual information related tothe user.

As yet another example, data storage engine 122 can receive data frommultiple data sources 110 (e.g., using one or more applicationprogramming interfaces) and data processing engine 124 can classify thereceived data as social data when the received data stream includesinformation related to social events involving multiple users (e.g., aconference scheduled using a social networking service, a calendarapplication, etc.), content and/or information published by one or moreusers using a service (e.g., a social networking service, a videosharing service, a photo sharing service, etc.), information about oneor more social connections of a user, and/or any other suitableinformation that can be classified as social data. In a more particularexample, social data associated with a user account of a social servicecan be retrieved in response to determining that the user account isalso authenticated on user device 102.

As still another example, data storage engine 122 can receive data frommultiple data sources 110 (e.g., using one or more applicationprogramming interfaces) and data processing engine 124 can classify thereceived data as personal data when the received data stream includesinformation about user goals, personal interests of a user (e.g., auser's stated interest available on a social networking service, mediacontent consumed and/or liked by a user, etc.), one or more utterancesgenerated by a user, and/or any other suitable information that can beregarded as personal. In this example, data processing engine 124 candiscard personal data unless specific authorization to use such personaldata is received from the user of user device 102.

In some implementations, data processing engine 124 can process datastreams that are provided by data source 110 and/or that are storedand/or processed by data storage engine 122.

In some implementations, data processing engine 124 can determinewhether data or a particular portion of data from data source 110 isrelevant to a goal or objective of the user. It should be noted that, insome implementations, data processing engine 124 can determine whetherdata or a particular portion of data from multiple data sources 110 isrelevant to a goal or objective of users assigned to a particular groupof users.

In some implementations, data processing engine 124 can determinewhether data or a particular portion of data from data source 110 isindicative of the emotional state of the user. These determinations canbe made in any suitable manner. For example, the determination can bemade using a suitable classifier that can classify input data or aportion of input data as being relevant to a goal or as being irrelevantto the goal.

In a more particular example, data processing engine 124 can select oneor more portions of data, where each portion of data can correspond toany suitable period of time, such as a few minutes, a couple of hours, aday of the week, a couple of days, a week, a month, etc. In someimplementations, the portions of data can be identified in any suitablemanner. For example, a determination can be made using a classifier thatcan classify a portion of data as being relevant to a goal. In anotherexample, a determination can be made using a classifier that canclassify a portion of data as likely to be indicative of a user'semotional state. In yet another example, a determination can be madeusing a classifier that can classify a portion of data as being relevantto a recommended action (e.g., data that can be used to determine thelikelihood that the action may impact the user's emotional state, datathat can be used to determine when the recommended action is to beexecuted, etc.). It should be noted that the classifier can be trainedusing any suitable machine learning algorithm, such as a support vectormachine, a decision tree, a Bayesian model, etc.

In some implementations, upon selecting various portions of data frommultiple data sources 110, data processing engine 124 can assign aweight to each of the portions of data. For example, for a particulargoal or objective, data processing engine 124 can determine that socialdata from particular data sources 110 is to be weighted such that it hasmore influence on the determination of the recommended action or output.This may be because social data that relates to the user and that isindicative of the user's emotional state is considered highly relevantto the objective of making new friends. In another example, this may bebecause social data tends to provide an accurate indication of theuser's emotional state (e.g., as the user of the user devices frequentlyposts status updates on multiple social networking websites) andbecause, prior to recommending a particular action, such as driving avehicle to a particular location, data processing engine 124 may takeinto account such social data. In another suitable example, weights canbe set by the user of user device 102 such that the user can tune theapplication and how particular types of data relating to the user areprocessed. In a more particular example, the user can set weightsassociated with social data such that the effect of social data in thedetermination of an action or output is reduced.

In some implementations, data processing engine 124 can generate one ormore profiles relating to the user. For example, data processing engine124 can use the received data to generate a baseline profile that isused to assign the user of user device 102 to a group (e.g., a group ofsimilar users for the particular objective or goal). In this example,data processing engine 124 can also generate a target profile for theindividual user and/or the group of users, which can include datacorresponding to similar users that have indicated an achievement of theparticular objective or goal. Alternatively, data processing engine 124can generate a target profile that includes data corresponding tosimilar users that have indicated a failure to attain the particularobjective or goal. As another example, data processing engine 124 canuse the received data and, in some implementations, request and receiveupdated data to generate a current profile associated with the user thatis indicative of the user's current physical or emotional state.

Any suitable profile relating to the user of user device 102 or a groupof users can be generated using any suitable approach. For example, dataprocessing engine 124 can generate one or more profiles that areindicative of the user's physical or emotional state over a given timeperiod. For example, a baseline profile associated with a user can begenerated based on data that is determined to be indicative of theuser's physical or emotional state during a given time period, such asmornings, a given day, weekdays, weekends, a given week, a season,and/or any other suitable time period. In another example, dataprocessing engine 124 can generate one or more profiles that areindicative of the user's physical or emotional state for a givencontext, such as a typical work day, a vacation day, mood and/orbehavior when user device 102 is located in the proximity of the user'shome, mood and/or behavior when user device 102 indicates that thetemperature in the proximity of user device 102 is below 65 degrees,etc.

In some implementations, server 120 can include an output recommendationengine 126 for determining and/or providing a recommended action thatmay affect or impact the physical or emotional state of the user. Forexample, in response to comparing a current profile corresponding to theuser with a target profile, output recommendation engine 126 candetermine a recommended action for the user. In a more particularexample, output recommendation engine 126 can, based on comparing thecurrent profile indicating that the user has a particular objective anda target profile of similar users that includes information relating tousers where it has been determined that they have achieved theparticular objective and information relating to users where it has beendetermined that they have not achieved the particular objective,determine one or more recommended actions that can impact the physicalor emotional state of the user and, upon performing the recommendedaction, may assist the user in reaching the particular objective.

It should be noted that, in some implementations, output recommendationengine 126 can cause any suitable recommended action to be executed onuser device 102 or any other suitable computing device associated withthe user. As shown in FIG. 2, the action or output can include, amongother things, a haptic or touch sensitive feedback, a sensory feedback(e.g., image content, light cues, music, video messages, video content,opening one or more relevant documents, etc.), an ambient-relatedfeedback (e.g., causing a scent to be emitted from a suitable device,modifying a lighting scheme by a lighting or home automation system,etc.), and/or a content-related action (e.g., presenting text, imagecontent, video content, audio content).

For example, output recommendation engine 126 can determine that amessage is to be presented to user device 102 associated with the userto prompt the user to engage in an activity. This may, for example,assist the user in reaching a particular objective or goal. In a moreparticular example, output recommendation engine 126 can determine thatthe message is to be presented in a particular form (e.g., by email,text message, mobile notification, account notification, a userinterface, and/or in any other suitable manner) and/or at a particulartime (e.g., based on the current physical or emotional state of theuser).

As another example, output recommendation engine 126 can determine thatan atmosphere is to be created in the proximity of the user. This may,for example, assist the user in reaching a particular objective or goaland/or affect the determined emotional state of the user. In a moreparticular example, based on a determination of the user's currentphysical or emotional state, output recommendation engine 126 can causemusic content, a feed of news articles that have been designated asbeing positive content, and/or a feed of image content that have beendesigned as being amusing content to be presented on user device 102associated with the user. In another more particular example, outputrecommendation engine 126 can cause a sound effect (e.g., rain sounds)to be presented on a device having an audio output device that isconnected to the user device, can cause ambient light in the user'ssurroundings to be adjusted using a lighting system connected to theuser device, and/or can cause a scent to be emitted by actuating a scentgenerator in a proximity of the user using user device 102.

As yet another example, output recommendation engine 126 can identifyone or more documents that are related to a particular task, activityand/or common subject matter. In some implementations, relevantdocuments can be automatically presented to the user and/or documentsthat are currently being presented can be arranged for presentationbased on output recommendation engine 126 identifying the documents asbeing related to the particular task, activity and/or common subjectmatter. These and other examples are discussed in more detail below inconnection with FIGS. 12-16.

Referring to FIG. 1, any suitable user device 102 can be used to executea recommended action from output recommendation engine 126. For example,user device 102 can be a wearable computing device, a television, amonitor, a liquid crystal display, a three-dimensional display, atouchscreen, a simulated touch screen, a gaming system, a portable DVDplayer, a portable gaming device, a mobile phone, a personal digitalassistant (PDA), a music player, a tablet, a laptop computer, a desktopcomputer, a mobile phone, a media player, a lighting device, a scentgenerator, and/or any other suitable device that can be used to performone or more recommended actions. It should be noted that, in someimplementations, user device 102 can have an application programminginterface such that the recommended output determined by outputrecommendation engine 126 can be transmitted to a suitable system, suchas a home automation system, where the system uses the applicationprogramming interface to cause the recommended output to be executed onone or more user devices 102.

In a more particular example, server 120 can determine that a userassociated with a user device has a particular objective or goal (e.g.,getting more exercise during the course of the user's workday). Inresponse to receiving authorization from the user of user device 102 toaccess social networking data, location data, and calendar data fromvarious devices and other data sources, server 120 can determine thatthe user of the user device is currently feeling relatively low energybased on the social data and that the user has a meeting that isscheduled at a particular time and that is taking place at a particularlocation from the calendar data (with no obligations between the currenttime and the time of the meeting). Server 120 can use such data and takeinto account historical data. For example, based on biometric data froma wearable pedometer associated with the user, server 120 can determinethe amount of activity that the user of user device 102 has engaged inthat month to date or week to date and determine whether the user islikely to meet an indicated objective or goal or likely to meet anaverage activity level. In another example, based on locationinformation, server 120 can determine the frequency that the user uses acar service to attend meetings at a particular location that is tenblocks away from a work location associated with the user. In yetanother example, based on stated interests and/or affinities on a socialnetworking service, server 120 can determine that the user of userdevice 102 likes flowers. In a further example, using mapping data thatdetermines a route between the work location associated with the userand the location of the meeting. Taking into account these portions ofdata from multiple devices and/or data sources, server 120 can cause oneor more recommended actions to be executed on one or more devices, suchas a notification to a user device that prompts the user to purchase acup of coffee from a nearby coffee shop in five minutes, a notificationto a user device that prompts the user to walk to the meeting using aparticular route that includes an option to visit an orchid shop thatrecently opened in a location that is along the provided route.Alternatively, server 120 can, at a particular time prior to themeeting, cause a scent generator located in proximity of user device 102to emit a lavender scent. In another alternative example, server 120can, at a particular time prior to the meeting, determine the weather inproximity of user device 102 prior to causing a notification thatprompts the user to walk to the meeting using a particular walk (e.g.,upon determining that the chance of precipitation is greater than aparticular threshold value, upon determining that it is “too hot” forthe user based on the determined temperature and user data as to what isconsidered “too hot,” etc.).

Continuing with this example, server 120 can determine that the user ofthe user device has visited the orchid shop and/or that the user iswalking to the coffee shop as recommended by the recommended action andserver 120 can use an application programming interface of the coffeeshop to request the number of consumers in the coffee shop and candetermine that the user may have a particular waiting time at the coffeeshop. Server 120 can then determine, using its respective applicationprogramming interface, that another coffee shop within the samefranchise has a lesser waiting time and is close to the user of userdevice 102 (e.g., a block away from the current location provided byuser device 102). Server 120 can transmit an updated or revisedrecommended action to user device 102.

In some implementations, it should be noted that server 120 can identifyone or more user devices 102 or other suitable devices for executing arecommended action that are in a particular proximity of the user (e.g.,a television, an audio system, a media player, a scent generator, alighting system, etc.). For example, server 120 can cause user device102 to detect devices that are connected to user device 102 and detectdevices that are in proximity of user device 102 (e.g., using devicediscovery functions). In response, server 120 can cause a song that isdeemed to be a relaxing song to be streamed from a service (e.g., amedia streaming service) and output using a device (e.g., a mobilephone, a media player, etc.) associated with the user. In addition,server 120 can cause a lavender scent to be emitted using a scentgenerator at a particular time in response determining that the userlikes lavenders (e.g., based on information published on the user'ssocial network page) and based on the current emotional state of theuser.

In some implementations, server 120 can personalize services formultiple users that each have a corresponding user device based on thecombined physical or emotional state of the users. For example, theusers can be a group of users having user devices that are in the samelocation (e.g., a coffee shop, a conference room, proximity of a givenuser, a town, an office, etc. based on location information or an onlinecalendar), a group of users having user devices that are connected toeach other on a social networking service, a group of users having userdevices that are determined to be similar users, and/or any othersuitable users.

Referring back to FIG. 1, system 100 can include one or more servers120. Server 120 can be any suitable server or servers for providingaccess to the mechanisms described herein for personalizing servicesbased on mood and/or behavior information from multiple data sources,such as a processor, a computer, a data processing device, or anysuitable combination of such devices. For example, the mechanisms forpersonalizing services based on mood and/or behavior information frommultiple data sources can be distributed into multiple backendcomponents and multiple frontend components and/or user interfaces. In amore particular example, backend components (such as mechanisms foridentifying an objective for a user, selecting particular portions ofdata from one or more data streams, generating profile information,determining recommended actions for one or more devices associated withthe user, etc.) can be performed on one or more servers 120. In anothermore particular example, frontend components (such as presentation of arecommended action in the form of content, executing a recommendedaction, detecting that a user device is near other devices, etc.) can beperformed on one or more user devices 102 and/or display devices 110.

In some implementations, each of user devices 102, data sources 110 andserver 120 can be any of a general purpose device, such as a computer,or a special purpose device, such as a client, a server, etc. Any ofthese general or special purpose devices can include any suitablecomponents such as a hardware processor (which can be a microprocessor,a digital signal processor, a controller, etc.), memory, communicationinterfaces, display controllers, input devices, etc. For example, userdevice 102 can be implemented as a smartphone, a tablet computer, awearable computer, a vehicle computing and/or entertainment system(e.g., as used in a car, a boat, an airplane, or any other suitablevehicle), a laptop computer, a portable game console, a television, aset-top box, a digital media receiver, a game console, a thermostat, ahome automation system, an appliance, any other suitable computingdevice, or any suitable combination thereof

Communications network 106 can be any suitable computer network orcombination of such networks including the Internet, an intranet, awide-area network (WAN), a local-area network (LAN), a wireless network,a Wi-Fi network, a digital subscriber line (DSL) network, a frame relaynetwork, an asynchronous transfer mode (ATM) network, a virtual privatenetwork (VPN), a peer-to-peer connection, etc. Each of communicationslinks 104, 108, and 112 can be any communications links suitable forcommunicating data among user devices 102, data sources 110, and server120, such as network links, dial-up links, wireless links, hard-wiredlinks, any other suitable communications links, or any suitablecombination of such links. Note that, in some implementations, multipleservers 120 can be used to provide access to different mechanismsassociated with the mechanisms described herein for personalizingservices based on mood and/or behavior information from multiple datasources. For example, system 100 can include: a data selection server120 that facilitates the selection of data from multiple data sourcesthat is indicative of an emotional state of the user; a profile server120 that generates a baseline profile to assign the user into a group ofusers, determines a target profile based on the assigned group of userand based on the objectives or goals of the user, generates a currentprofile representing the user, and compares the current profile with thetarget profile; a recommendation server 120 that determines one or morerecommended actions that may have a likelihood of impacting theemotional state of the user and/or may move the user towards anobjective or goal; a delivery server 120 that causes the recommendedaction to be executed (e.g., transmit content to a particular device,transmit instructions to a home automation system, etc.); and/or anyother suitable servers for performing any suitable functions of themechanisms described herein.

FIG. 3 shows an example 300 of hardware that can be used to implementone or more of user devices 102 and servers 120 depicted in FIG. 1 inaccordance with some implementations of the disclosed subject matter.Referring to FIG. 3, user device 102 can include a hardware processor302, a display/input device 304, memory 306, and a transmitter/receiver308, which can be interconnected. In some implementations, memory 306can include a storage device (such as a computer-readable medium) forstoring a user device program for controlling hardware processor 302.

Hardware processor 302 can use the user device program to execute and/orinteract with the mechanisms described herein for personalizing servicesbased on mood and/or behavior using multiple data sources, which caninclude presenting one or more recommendation interfaces (e.g., forinputting objective or goal information, for providing authorization toaccess data from one or more data sources, for selecting data sources,etc.), and can include executing a recommended action. In someimplementations, hardware processor 302 can transmit and receive datathrough communications link 104 or any other communication links using,for example, a transmitter, a receiver, a transmitter/receiver, atransceiver, and/or any other suitable communication device, such astransmitter/receiver 308. Display/input device 304 can include atouchscreen, a flat panel display, a cathode ray tube display, aprojector, a speaker or speakers, and/or any other suitable displayand/or presentation devices, and/or can include a computer keyboard, acomputer mouse, one or more physical buttons, a microphone, a touchpad,a voice recognition circuit, a touch interface of a touchscreen, acamera, a motion sensor such as an optical motion sensor and/or anaccelerometer, a temperature sensor, a near field communication sensor,a biometric data sensor, and/or any other suitable input device.Transmitter/receiver 308 can include any suitable transmitter and/orreceiver for transmitting and/or receiving, among other things,instructions for presenting content, information related to a currentcontrol level, requests for location information, etc., and can includeany suitable hardware, firmware and/or software for interfacing with oneor more communication networks, such as network 106 shown in FIG. 1. Forexample, transmitter/receiver 308 can include: network interface cardcircuitry, wireless communication circuitry, and/or any other suitabletype of communication network circuitry; one or more antennas; and/orany other suitable hardware, firmware and/or software for transmittingand/or receiving signals.

Server 120 can include a hardware processor 312, a display/input device314, memory 316 and a transmitter /receiver 318, which can beinterconnected. In some implementations, memory 316 can include astorage device (such as a computer-readable medium) for storing arecommendation program for controlling hardware processor 312.

Hardware processor 312 can use the recommendation program to executeand/or interact with the mechanisms described herein for: obtaininginformation associated with an objective of a user of a computing devicefrom a plurality of data sources; identifying an objective for a user ofa user device; receiving information associated with the user frommultiple data sources; determining that a portion of information fromeach of the multiple data sources is relevant to the user having theidentified objective; assigning the user into a group of users from aplurality of groups based on the identified objective and the portion ofinformation from each of the multiple data sources; determining a targetprofile associated with the user based on the identified objective andthe assigned group; generating a current profile for the user based onthe portion of information from each of the multiple data sources;comparing the current profile with the target profile to determine arecommended action, where the recommended action is determined to have alikelihood of impacting the emotional state of the user; causing therecommended action to be executed (e.g., on a device possessed by, orlocated proximate to, the user); determining one or more devicesconnected to the computing device, wherein each of the one or moredevices has one or more device capabilities; and/or transmitting andreceiving data through communications link 108. In some implementations,the recommendation program can cause hardware processor 312 to, forexample, execute at least a portion of process 400 as described below inconnection with FIG. 4. In some implementations, hardware processor 312can transmit and receive data through communications link 114 or anyother communication links using, for example, a transmitter, a receiver,a transmitter/receiver, a transceiver, and/or any other suitablecommunication device such as transmitter/receiver 318. Display/inputdevice 314 can include a touchscreen, a flat panel display, a cathoderay tube display, a projector, a speaker or speakers, and/or any othersuitable display and/or presentation devices, and/or can include acomputer keyboard, a computer mouse, one or more physical buttons, amicrophone, a touchpad, a voice recognition circuit, a touch interfaceof a touchscreen, a camera, a motion sensor such as an optical motionsensor and/or an accelerometer, a temperature sensor, a near fieldcommunication sensor, a biometric data sensor, and/or any other suitableinput device. Transmitter/receiver 318 can include any suitabletransmitter and/or receiver for transmitting and/or receiving, amongother things, content to be presented, requests for status informationof display device 110, requests for content, requests for locationinformation, etc., and can include any suitable hardware, firmwareand/or software for interfacing with one or more communication networks,such as network 106 shown in FIG. 1. For example, transmitter/receiver318 can include: network interface card circuitry, wirelesscommunication circuitry, and/or any other suitable type of communicationnetwork circuitry; one or more antennas; and/or any other suitablehardware, firmware and/or software for transmitting and/or receivingsignals.

In some implementations, server 120 can be implemented in one server orcan be distributed as any suitable number of servers. For example,multiple servers 120 can be implemented in various locations to increasereliability and/or increase the speed at which the server cancommunicate with user devices 102 and/or data sources 110. Additionallyor alternatively, as described above in connection with FIG. 1, multipleservers 120 can be implemented to perform different tasks associatedwith the mechanisms described herein.

Turning to FIG. 4, an illustrative example 400 of a process forpersonalizing computerized services based on the physical or emotionalstate of a user of a user device using data from multiple data sourcesin accordance with some implementations of the disclosed subject matteris shown.

It should be noted that process 400 can personalize computerizedservices, where data from multiple data sources can be used to determinethe impact of a computerized service on a physical or emotional state ofa user having a user device. The emotional state of a user can be acomplex phenomenon. Emotion can be a mental state that is associatedwith physiological activity and can be influenced by internal orexternal environmental conditions. Emotion can be associated withpersonality, mood, temperament, disposition, and motivation of a user.For example, emotional states can include happiness, contentment,tranquility, surprise, anger, fear, sadness, depression, disgust, tired,anxious, hurried, etc. In some examples, emotional states can be broadlyclassified into positive and negative emotions, where positive emotionscan include happiness and contentment and negative emotions can includeanger and depression. In addition, an example of an internalenvironmental condition includes an old memory and an example ofexternal stimulus includes stress or the relief of stress due to variousenvironmental factors.

It should also be noted that the physical or emotional state of a usercan be considered an overall snapshot or view of the user's physicalcharacteristics or emotions at a point in time. Because multiple factorscan be involved in a user's physical or emotional state, the physical oremotional state can fluctuate even over short periods of time. By usingdata relating to the user from multiple sources, a user's physical oremotional state can be predicted, which can be used to determine whetherto recommend a particular computerized action at a given time. Moreover,changes to a user's physical or emotional state can be predicted basedon new or updated data relating to the user from multiple sources. Evenfurther, changes to a user's physical or emotional state can be used toevaluate whether recommended computerized actions to devices possessedby, or located proximate to, the user may be moving the user towards aparticular goal or objective.

As illustrated, process 400 can begin by receiving user input relatingto a particular objective or goal at 410. Illustrative examples of aparticular objective or goal can be getting more exercise (e.g.,generally increasing the current activity level, getting any form ofexercise for at least one hour per day, etc.), losing weight (e.g.,generally losing weight, losing ten pounds in three months, etc.),making more friends, accomplishing a particular emotional state (e.g.,feel more productive, feel less stressed, etc.), etc.

In a more particular example, in response to receiving authorizationfrom a user of a user device to access social data relating to the userfrom a social networking service, process 400 can extract keywords fromsocial media posts published by the user on the social networkingservice to determine one or more objectives of the user. In thisexample, social data relating to the user from a social networkingservice can be received, which can include messages or posts havingtext, image content, video content, and/or audio content, messagesposted by other users that are connected to the user, and contextualinformation, such as timing information, location information, and adeclared mood or emotional state of the user or users connected to theuser.

In another more particular example, in response to installing arecommendation application on a computing device associated with theuser, the recommendation application can present a recommendationinterface on the computing device that prompts the user to select anobjective from the user interface. For example, the recommendationinterface can be presented as a recommendation card, a notification, orany other suitable user interface that prompts the user to indicate anobjective or goal. An illustrative example of a recommendation interfacethat can be presented on a computing device is shown in FIG. 5. Asshown, in some implementations, a user device 102, such as a mobiledevice 500, can prompt the user to input an objective in recommendationinterface 510, such as “get more exercise” or “improve your mood.” Thesuggested objectives in recommendation interface 510 can be presentedbased on any suitable criterion (e.g., default objectives, popularobjectives, objectives selected based on recent searches inputted intothe user device, objectives selected based on location informationassociated with the user device, objectives based on attributes inferredfrom data sources authorized by the user of the user device, etc.). Asalso shown, a reason for the suggested objective can be provided, suchas “Based on your recent searches” and “Based on your recent posts.”Additionally or alternatively, the recommendation interface can presentthe user of mobile device 500 with a search field to provide keywordsrelating to an objective or goal that the user desires to achieve.

Referring back to FIG. 4, at 420, the recommendation system candetermine one or more goals for a user of a user device based on thedetermined objective. For example, in response to determining that theobjective is to lose weight, the recommendation system can determinegoals that are associated with the objective of losing weight—e.g.,achieving a first activity level for the first week and a secondactivity level for the second week, achieving an average activity levelover the first month, waking up at a particular time every morning,achieving a threshold amount of rest at the end of each day, eating atparticular times on weekdays, etc. As described hereinbelow, therecommendation system can generate various profiles, such as profiles ofsimilar users that each have user devices, profiles of users having userdevices who have indicated that they have achieved the determinedobjective or one of the goals, profiles of users having user devices whohave indicated that they have failed to achieve the determined objectiveor one of the goals, etc. In this example, the recommendation system canprocess these profiles to determine goals associated with an objective(e.g., which goals were achieved by users that are deemed to be similarto the user, which goals were achieved within a particular amount oftime, etc.). In a more particular example, in response to selecting oneof the objectives presented in recommendation interface 510, therecommendation system can determine multiple goals associated with theselected objective and select a portion of those goals based on profileinformation.

In some implementations, the recommendation system can receive anysuitable data associated with the user from multiple data sources at430. For example, from one or more data sources, the recommendationsystem can receive and/or request data relating to activities engaged inby one or more users of user devices, such as took a walk and thedistance traversed using a mobile device with location services, visiteda location that corresponds with a coffee shop using a mobile devicewith social services, attended a social event (e.g., a conference, aparty, a sporting event, etc.) using a mobile device with an onlinecalendar, attended a fitness training session using a mobile device withan online calendar and/or social services, etc. As another example, fromone or more data sources, the recommendation system can receive and/orrequest data that includes timing information related to an activity,such as a duration of the activity, a time corresponding to theactivity, etc. As yet another example, the recommendation system canreceive and/or request data that includes a number of occurrences of anactivity engaged in by one or more users during a given time period(e.g., a day of the week, a couple of days, weekdays, weekends, etc.), anumber of users that engage in a given activity, and/or any othersuitable information relating to frequency information related to agiven activity.

In some implementations, the recommendation system can receive and/orrequest data relating to content and/or information published by theuser on a social networking service. For example, the data can includeone or more mood states published by a user on a service (e.g., a socialnetworking service, a messaging service, a video sharing service, aphoto sharing service, an electronic commerce service, etc.). As anotherexample, the data can include comments, messages, posts, locations,and/or any other suitable content published by the user on a socialnetworking service. As still another example, the data can include anysuitable information relating to one or more social connections of theuser on a social networking service, content posted by the socialconnections, locations associated with the social connections, etc.

In some implementations, the recommendation system can receive and/orrequest data relating to user interactions with one or more mediacontent items. For example, the data can include any suitableinformation relating to a media content item with which the user hasinteracted. In a more particular example, the data can include a type ofthe media content item, a description of the media content item, a linkto the media content item (e.g., a URL), an identifier that can identifythe media content item (e.g., a URI, a program identifier, etc.), anauthor of the media content item, an artist related to the media contentitem, etc. As another example, the data can include any suitableinformation about a type of a user interaction with a media content itemon a user device, such as consuming the media content item, publishingthe media content item via a social networking service or any othersuitable service, sharing the media content item with other users,liking the media content item via a social networking service or anyother suitable service, commenting on the media content item, etc. Asyet another example, the data can include any suitable timinginformation related to a user interaction with a media content item on auser device, such as a duration of the user interaction, a timecorresponding to the user interaction, etc.

In some implementations, the recommendation system can receive and/orrequest biometric data associated with the user of a user device. Forexample, in response to receiving authorization to access biometric datafrom a data source that includes a sensor, the biometric data caninclude any suitable physiological parameter associated with the user,such as a heart rate, a respiration rate, a blood pressure, a bodytemperature, skin moisture, etc. As another example, the biometric datacan include a range of physiological parameters, such as a heart raterange, a blood pressure range, etc.

In some implementations, the recommendation system can receive and/orrequest location data associated with the user of a user device. Forexample, in response to receiving authorization to access locationinformation, the location data can include any suitable information thatcan be used to estimate a location of a computing device associated withthe user, such as an identifier associated with the computing device(e.g., an IP address, a device identifier, a media address control (MAC)address, a serial number, a product identifier, etc.), GPS signalsgenerated by the computing device, Wi-Fi access points associated withthe computing device, information about a cell tower to which thecomputing device is connected, etc. As another example, the locationdata can include any suitable information that can be used to estimate alocation of the user, such as a location published by the user using asuitable service (e.g., a social networking service), a location that auser intends to visit (e.g., a location associated with a social eventscheduled using a calendar application executing on a mobile device, asocial network account associated with the user, etc.), etc.

In some implementations, the recommendation system can present arecommendation interface, such as the recommendation interface shown inFIG. 6, where the user of mobile device 500 is prompted with datasources for selection. For example, various data sources can be detectedby the recommendation system executing on mobile device 500 and, inresponse to detecting the various data sources, the user can be promptedto select which data sources to obtain data associated with the user. Asshown in FIG. 6, a recommendation interface prompts the user of mobiledevice 500 to select from various data sources that are available to therecommendation application, where the user has indicated a permission toaccess location data from mobile device 500 and social data fromservices that have been authenticated using mobile device 500. In a moreparticular example, the recommendation system can prompt the user toprovide authorization to access particular data sources and select whichdata sources may include data that is relevant towards accomplishing agoal or objective. In this example, the recommendation system canprovide an interface prompting the user of the user of the user deviceto provide credentials, such as a username and password, for accessing aparticular data source.

In some implementations, the recommendation system can prompt the userto provide additional information in response to selecting one or moredata sources for obtaining user, such as using recommendation interface610. For example, the recommendation system can determine that, in orderto generate a baseline profile for the user, certain portions of thebaseline profile can be derived or satisfied using data obtained fromthe selected data sources and other portions of the baseline profileremain incomplete. In response, the recommendation system can generatean interface that prompts the user to provide such information—e.g., ifthe goal is “losing weight,” such an interface can prompt the user ofmobile device 500 to input a height value and a weight value.

Referring back to FIG. 4, the recommendation system can select portionsof the data received from multiple data sources based on the objectivesor determined goals at 440. For example, the recommendation system canreceive data from multiple data sources (e.g., using one or moreapplication programming interfaces) and can determine that the receiveddata is to be classified into various categories of data. Thesecategories can include, for example, general data, contextual data,social data, and personal data. Examples of general data can includeinformation about one or more services used by a user (e.g., a socialnetworking service, an email service, a messaging service, a videosharing service, etc.), search history associated with a user (e.g.,keywords inputted by the user), etc. Examples of contextual data caninclude information about a location of user device 102, trafficinformation, weather information based on location information from userdevice 102 (e.g., “sunny,” “cold,” etc.), population density informationwithin a given location, a location context relating to data provided bya data source 110 (e.g., “work,” “home,” “vacation,” etc.), informationrelating to devices located near or connected to a user device, and/orany other suitable information that can provide contextual informationrelated to the user. Examples of social data can include informationrelated to social events involving multiple users (e.g., a conferencescheduled using a social networking service, a calendar application,etc.), content and/or information published by one or more users using aservice (e.g., a social networking service, a video sharing service, aphoto sharing service, etc.), information about one or more socialconnections of a user, and/or any other suitable information that can beclassified as social data. Examples of personal data can includepersonal interests of a user (e.g., a user's stated interest availableon a social networking service, media content consumed and/or liked by auser, etc.), one or more utterances generated by a user, and/or anyother suitable information that can be regarded as personal.

In some implementations, the recommendation system can create a datastream for each category of data. For example, in response tocategorizing particular data from multiple services as being socialdata, the recommendation system can aggregate the social data as it isbeing received and create a social data stream that includes timestampedsocial data from the multiple sources. Alternatively, upon receivingauthorization from the user to access a particular data source, therecommendation system can categorize the data received from that sourceand place the data into a data stream that is associated with that datasource, such as a social data stream of timestamped social data from aparticular social source. For example, as shown in FIG. 7, multiple datastreams from multiple data sources can be obtained−e.g., general data(G5 and G13), personal data (P1 and P42), social data (S9 and S25), andcontextual data (C33 and C57).

In some implementations, the recommendation system can select particularportions of data by determining which categories of data to analyze andwhich portions of the data are to be used to determine a recommendationaction that may, for example, affect the physical or emotional state ofthe user. In response to determining an objective at 410 or a goal at420 for the user of the user device, the recommendation system canselect particular categories of data that may include data that isrelevant to the objective or goal. For example, the recommendationsystem can determine that social data and contextual data are likely tobe relevant to the objective of losing weight. In response to analyzingthe data relating to the user from multiple data sources at 430, therecommendation system can select particular categories of data fromparticular data sources and select particular time portions of data thatare indicative or representative of the physical or emotional state ofthe user. For example, the recommendation system can, in response toreceiving authorization from a user of a user device to receive datarelating to the user from multiple data sources, determine that, duringweekdays between 9AM and 5PM, the user device is not typically used onsocial data sources and that contextual data from the user device anddevices connected to the user device are likely to be representative ofthe emotional state of the user. It should be noted that, using thereceived data and/or the determined objectives and goals, therecommendation system can select different subsets of data for makingdifferent determinations—e.g., a subset of data for recommending aparticular action, a subset of data that is indicative of the emotionalstate of the user during a particular time of the day, a subset of datathat is indicative of the emotional state of the user during an averageday, a subset of data that is representative of the activities of theuser on a given day, etc.

In some implementations, each objective or goal can have an associateddata template for retrieving data that is related to the user and thatis relevant to the objective or goal. For example, in response todetermining an objective at 410 or a goal at 420, the recommendationsystem can retrieve an associated data template that includes particulardata fields, such as particular social-related data fields (e.g.,keywords extracted from social posts and an associated time),contextual-related data fields (e.g., location information from multipledevices associated with the user corresponding to the times of eachsocial post), and general data fields (e.g., type of applications thatthe user device has installed and device profiles of devices that arenearby the user device). As described above, in response to determiningthat information for particular data fields may not be completed orderived using data from the data sources, the recommendation system canprompt the user to input such missing data (e.g., by generating a userinterface prompt the user to input data and/or input the accuracy ofinferences made about the user).

It should be noted that, although the recommendation system can make adetermination based on particular subsets of data and can retrieve datatemplates that request particular portions of data, the user of a userdevice, such as user device 102, can be provided with controls forsetting which data sources are used (e.g., a specific social networkingservice, a specific mobile device, etc.) and which types of data areused by the recommendation system (e.g., social information from aspecific social networking service and not data determined to includepersonal information, social post information from a social networkingservice and not relationship information from a social messagingservice, etc.). For example, the user can be provided with anopportunity to select a particular type of data from a particular datasource that may include data relevant to the user for a particular goalor objective.

In some implementations, using the selected portions of data from themultiple data sources, the recommendation system can determine abaseline profile for the user at 450. For example, the recommendationsystem can process the selected portions of data and generate one ormore baseline profiles associated with each objective or goal. In a moreparticular example, a baseline user profile associated with a goal caninclude any suitable information relating to the physical or emotionalstate of the user (e.g., “happy,” “unhappy,” etc.) and information aboutone or more user behaviors or habits (e.g., commuting, lunch break,weekly meetings, exercise groups, etc.). In another more particularexample, a baseline user profile can use heart rate information,temperature information, galvanic skin response information, locationinformation, and social post information, match such information with anemotional state, and establish baseline patterns for emotional statethrough a given time period, a day, a week, a season, etc.

In some implementations, using the selected portions of data from themultiple data sources, the recommendation system can determine anoverall baseline profile for the user that includes multiplesub-profiles—e.g., a sub-profile that uses the data to predict thecurrent emotional state of the user, a sub-profile that describes thetypical activity level of the user, a sub-profile that describes typicalbehaviors of the user at particular times of the day, etc. Any suitablenumber of sub-profiles can be generated to create an overall baselineprofile of the user. It should also be noted that, in someimplementations, the recommendation system can use different subsets ofdata for each of the sub-profiles that form the overall baseline profileof the user.

In some implementations, the recommendation system can assign the userto a group of users based on the baseline profile at 450. For example,the recommendation system can identify a group of users that haveaccomplished the goal or objective and one or more behaviors associatedwith users in the group and/or actions performed by users in the group.In another example, the recommendation system can identify a group ofusers that have failed to accomplish the goal or objective and one ormore user behaviors associated with users in the group and/or actionsperformed by users in the group. The recommendation system can thencorrelate particular behaviors and/or actions with the goal for theuser.

In some implementations, the recommendation system can use machinelearning techniques to identify and cluster similar user profiles. Forexample, the recommendation system can use machine learning techniquesto determine which group profile is most similar to the baseline profileassociated with the user and, in response, can place the user into thegroup associated with that group profile. In another example, therecommendation system can use machine learning techniques to determinewhich group profile includes users having user devices that are similarto the user of the user device and includes users interested inattaining the same objective. In yet another example, the recommendationsystem can use machine learning techniques to determine which groupprofile has sub-profiles that include common features to thesub-profiles that form the overall baseline profile of the user. Itshould be noted that any suitable machine learning technique can beused, such as a support vector machine, a decision tree, a Bayesianmodel, etc.

It should be noted that, in some implementations, other information canbe used to group similar users together. For example, the group of userscan include users having user devices that are in a similar geographicproximity, such as users that are in the same city as a particular user.As another example, the group of users can include users that areconnected to each other on one or more social networking services.

It should also be noted that, in some implementations, process 400 canreturn to 420, where the recommendation system can determine one or moregoals for achieving a particular objective based on the assigned groupof users. For example, for a particular objective, the recommendationsystem can retrieve one or more goals that are associated with anassigned group of similar users that have indicated a desire to reachthe objective. In another example, for a particular objective, therecommendation system can rank the goals associated with an objective,where the ranking is based on inputs from users in the group of users asto which goals assisted the user in reaching the objective. Therecommendation system can then select at least a portion of the goalsfor the user that may assist the user in reaching the objective. Theselected goals can then be presented to the user in a recommendationinterface, such as recommendation interface 800 shown in FIG. 8. In thisexample, the recommendation system can provide the user of mobile device500 with an opportunity to remove and/or add additional goals.

In some implementations, the recommendation system can use the baselineprofile generated at 450 for other determinations. For example, therecommendation system can determine whether a current profile thatincludes updated data relating to the user from multiple data sourcesdeviates from the previously generated baseline profile. Deviationsbetween the baseline profile and the current profile can include, forexample, a comparison of the frequency of particular activities (e.g.,exercise frequency) and a comparison of the timing information relatingto particular behaviors (e.g., the time when the user wakes up eachday). Such a deviation can indicate that the data or such determinationsbased on the data may not be indicative of the emotional state of theuser (e.g., a stress response may be detected from the user in responseto a job change). Such a deviation can also indicate that therecommendation system is to update the baseline profile and/or updatethe assignment of the user into another group of users (e.g., as theuser is progressing towards a goal or objective, as the behaviors of theuser have changed over time, etc.). In another example, such a deviationcan indicate that the recommendation system is to recommend actions thatmay return the user back to the baseline profile.

In a more particular example, the baseline profile generated by therecommendation system can include behaviors and/or activities associatedwith the user (e.g., consuming classical music, attending a fitnesssession, etc.), timing information relating to each of the behaviorsand/or activities (e.g., time spent listening to classical music),frequency of a particular behavior and/or activity over a given timeperiod (e.g., the number of times the user using the user device haslistened to classical music during the week), threshold valuesassociated with behaviors and/or activities (e.g., the user tends tolisten to classical music at least three times a week for at leastthirty minutes each session), etc.

In another more particular example, the baseline profile generated bythe recommendation system can include any suitable representation ofdata related to the user. For example, in response to receiving aparticular portion of biometric data, the recommendation system candetermine an average heart rate for the user while at the office, anaverage number of calories burned on weekdays, and an activity curve foran average day for the user.

It should also be noted that multiple baseline profiles can be generatedand associated with the user of the user device. For example, therecommendation system can generate a baseline profile using a firstsubset of data that is associated with a goal (e.g., getting at leastthirty minutes of exercise per day) and another baseline profile using asecond subset of data that is associated with another goal (e.g., usingan email application for less than a particular amount of time). Inanother example, the recommendation system can generate a baselineprofile in a particular context, such as “work,” and another baselineprofile in another context, such as “vacation.”

In some implementations, the recommendation system can generate a targetprofile based on the assigned group, the goals, and/or the objective at470. For example, for a particular objective, the recommendation systemcan identify and cluster user profiles of users where it has beendetermined that the user has met a goal or an objective. In anotherexample, the recommendation system can identify and cluster userprofiles of users where it has been determined that the user has not meta goal or an objective (e.g., to determine which actions may not beassisting users in meeting a particular goal or objective). In yetanother example, the recommendation system can identify and cluster userprofiles of users that the recommendation system has previously assistedthe users in attaining a stated goal or objective.

In a more particular example, the recommendation system can generate atarget profile for achieving a particular goal or objective using aprofile that includes information relating to users that have met theparticular goal or objective and information relating to users that havenot met the particular goal or objective. In this example, therecommendation system can determine actions, threshold values, and otherinformation that may assist the user in attaining the particular goal orobjective—e.g., users that have been determined to achieve the objectiveof losing ten pounds in a month have also walked at least one mile eachday, woken up by 6 AM in the morning, listened to classical music in theevening, and eaten meals at particular times. By, for example,determining common features between users that have indicated anachievement of the particular goal or objective, the recommendationsystem can generate a target profile that can be used to recommendactions to the user. These actions, if performed, may affect the currentprofile of the user such that the current profile of the user movestowards the target profile.

Referring back to FIG. 4, in some implementations, the recommendationsystem can generate a current profile for the user based on updated datafrom the multiple data sources at 480. It should be noted that thebaseline profile and the current profile associated with the user can bedynamic profiles that can be generated using updated data from themultiple data sources. For example, in response to determining that aparticular period of time has elapsed (e.g., one minute, one day, etc.),the recommendation system can receive and/or request updated data fromthe multiple data sources and generate a current profile for the user.Alternatively, the recommendation system can continue to use thebaseline profile.

In some implementations, the recommendation system can compare thecurrent profile with the target profile to determine a recommendedaction at 490. This may, for example, impact the physical or emotionalstate of the user. Based on the objective or goal and the profileinformation, the recommendation system can determine which computerizedaction is to be executed at the user device, a device that the userpossesses, or a device that is proximate to the user device.

In some implementations, the recommendation system can determinemultiple computerized actions that are recommended to the user of theuser device at various times. For example, the recommendation system candetermine that a user has the particular objective of getting moreexercise during the course of the user's workday. In response toreceiving authorization from the user of a user device to access socialnetworking data from a social networking service, location data from amobile device associated with the user, and calendar data from an onlinecalendar associated with the user, the recommendation system candetermine that the user is currently feeling a relatively low energyfrom the social data and that the user has a meeting that is scheduledat a particular time and that is taking place at a particular locationfrom the calendar data (with no obligations between the current time andthe time of the meeting). The recommendation system can use such dataand incorporate other data into a dynamic user profile. For example,based on biometric data from a wearable pedometer associated with theuser, the recommendation system can determine the amount of activitythat the user has engaged in that month to date or week to date anddetermine whether the user is likely to meet an indicated objective orgoal or likely to meet an average activity level. In another example,based on location information, the recommendation system can determinethe frequency that the user uses a car service to attend meetings at aparticular location that is ten blocks away from a work locationassociated with the user. In yet another example, based on statedinterests and/or affinities on a social networking service, therecommendation system can determine that the user of the user devicelikes flowers. In a further example, using mapping data, therecommendation system can determine a route between the work locationassociated with the user and the location of the meeting. Taking intoaccount these portions of data from multiple data sources, therecommendation system can generate a current profile associated with theuser and compare it with a target profile that can be associated withthe particular objective and/or with a particular group of users. Basedon the comparison, the recommendation system can cause one or morerecommended actions to be executed, such as a notification that promptsthe user to purchase a cup of coffee from a nearby coffee shop in fiveminutes, a notification that prompts the user to walk to the meetingusing a particular route that includes an option to visit an orchid shopthat recently opened in a location that is along the provided route.Alternatively, the recommendation system can, a particular time prior tothe meeting, cause a scent generator located in proximity of the userdevice to emit a lavender scent. In another alternative example, therecommendation system, at a particular time prior to the meeting,determine the weather in proximity of the user device prior to causing anotification that prompts the user to walk to the meeting using aparticular route as identified by a computer map route service, such asGoogle Maps (e.g., upon determining that the chance of precipitation isgreater than a particular threshold value, upon determining that it is“too hot” for the user based on the determined temperature and user dataas to what is considered “too hot,” etc.).

Continuing with this example, the recommendation system can determinethat the user has visited the orchid shop and/or that the user iswalking to the coffee shop as recommended by the recommended action. Therecommendation system can then use an application programming interfacecorresponding to the coffee shop to request the number of consumers inthe coffee shop and can determine that the user may have a particularwaiting time at the coffee shop. The recommendation system can, usingits respective application programming interface, then determine thatanother coffee shop within the same franchise has a lesser waiting timeand is nearby the user of the user device (e.g., a block away from thecurrent location provided by the user device). The recommendation systemcan transmit an updated or revised recommended action to the userdevice.

It should be noted that each of these multiple computerized actions canbe associated with a corresponding trigger event. For example, anaction, such as the notification prompting the user to purchase coffeefrom a nearby coffee shop, can be triggered based on an associated time(e.g., time of day, time of the preceding event, time until the nextscheduled event begins, etc.). In another example, an action, such asthe notification prompting the user to visit an orchid shop along theroute to a scheduled meeting, can be triggered based on locationinformation associated with the user device (e.g., detecting that theuser device is within a particular proximity of the orchid shop). In yetanother example, the recommendation system can determine that the actionis a message that is to be presented in a particular form (e.g., byemail, text message, mobile notification, account notification, and/orin any other suitable manner) and/or at a particular time (e.g., basedon the predicted emotional state of the user).

Referring back to FIG. 4, the recommendation system can cause therecommended action or actions to be executed at 495. Illustrativeexamples of recommended actions are shown in FIG. 2. As shown, theaction or output can include, among other things, a haptic or touchsensitive feedback, a sensory feedback (e.g., image content, light cues,music, video messages, video content, etc.), an ambient-related feedback(e.g., causing a scent to be emitted from a suitable device, modifying alighting scheme by a lighting or home automation system, etc.), and/or acontent-related action (e.g., presenting text, image content, videocontent, audio content). In a more particular example, the recommendedaction can include modifying a sound, cancelling a sound, or enhancing asound in the background of the user of the user device using an audiooutput device that is connected to the user device. In another moreparticular example, the recommended action can include providing sensoryfeedback (e.g., light cues, audio cues, video cues, scent cues, etc.) inthe environment of the user of the user device to provide anotification. In yet another more particular example, the recommendedaction can include nostalgia-oriented feedback including content-relatedactions based on historical information relating to the user. In afurther example, the recommended action can include a prioritization ofapplication data based on device information and other informationrelating to the user (e.g., the organization of user interface elements,the positioning of documents or files, etc.).

In some implementations, the recommendation system can, based on therecommended action, identify one or more devices that may be connectedto, or proximate to, the user of user device for executing therecommended action. In some implementations, the recommendation systemcan initiate device discovery functions to determine which device ordevices are near the user device. In some implementations, such devicediscovery functions can be initiated in response to launching arecommendation application on the user device or in response todetermining that a recommended action is to be executed using a device.Additionally or alternatively, in some implementations, such devicediscovery functions can be initiated from any suitable device and canuse any suitable information to determine which devices are near theuser device.

In some implementations, the user device can determine whether anyoutput devices are nearby. The user device or the recommendationapplication executing on the user device can use any suitable techniqueor combination of techniques to determine whether any output devices arenearby. For example, the user device can transmit a signal or signalsincluding a message requesting that nearby devices (e.g., devices whichreceive the signal) to respond with a message indicating that the devicereceived the signal. In this example, the response can include anysuitable device information, such as device location information anddevice capability information. As another example, the user device canreceive a signal or signals transmitted by a device including a messageindicating that the display device is available for causing arecommended action or output to be executed. Such signals can betransmitted using, for example, peer-to-peer communication techniquessuch as Bluetooth, using RFID techniques, and/or using any othersuitable technique or combinations of techniques for communicatingbetween the user device and one or more output devices.

In some implementations, the recommendation system can provide the userwith the opportunity to select one or more output devices that areavailable for executing a recommended action. For example, as shown inFIG. 9, recommendation interface 910 can provide the user with theopportunity to select, add, or remove various output devices that havebeen detected as being connected to or proximate mobile device 500. Asshown, such output devices can include a television device, a homeautomation system, a tablet computing device, a scent generator, and anautomobile. In some implementations, the recommendation interface canprovide the user with the opportunity to request that the user devicedetect additional output devices (e.g., in response to moving to adifferent location that is in the proximity of other devices).

In some implementations, the recommendation system can cause an outputdevice to execute a particular action based on the physical or emotionalstate of the user. For example, prior to executing the particular actionusing the output device, the recommendation system can determine thecurrent emotional state of the user and, upon determining that theemotional state of the user is “angry” based on user data, can inhibitthe action from being executed on the output device. In another example,the recommendation system can determine that the particular action canbe executed on the output device upon determining that the emotionalstate of the user is anything except for “angry”—e.g., as therecommendation system has determined from historical user data thatactions taken by one or more output devices are not well received whenthe user is experiencing an “angry” emotional state.

Additionally or alternatively, in some implementations, therecommendation system can cause an output device to execute a particularaction based on the predicted impact of the particular action on thecurrent physical or emotional state of the user. For example, prior toexecuting a particular action using the output device, therecommendation system can determine the predicted impact of the actionon the physical or emotional state of the user and, upon determiningthat the predicted impact is not within a particular range (e.g., theemotional state correlated with the user data remains unchanged), caninhibit the action from being executed on the output device.

As shown in FIG. 10 and in connection with the above-mentioned example,the recommendation system can present the user with a recommendationinterface 1010 that includes multiple recommended actions. As shown,each recommended action that is presented in recommendation interface1010 can include additional information for performing the recommendedaction, such as a map of a route in response to recommending that theuser walk to the location of an event or commerce information inresponse to recommended that the user purchase a cup of coffee. As alsoshown, each recommended action can be associated with a particular time,such as purchasing a cup of coffee now or beginning a walk to the eventat 1:45 PM. In some implementations, as described above, eachrecommended action in recommendation interface 1010 can be triggered bythe occurrence of a particular event, such as a determination that theuser device is associated with a particular location, a determinationthat the user device indicates the user is walking along a particularroute, etc.

It should be noted that, in some implementations, a recommended actioncan be executed for multiple users. For example, as described above, therecommendation system can place the user of the user device into a groupof users having a similar baseline profile. In addition, therecommendation system can place the user into a group of users based onother suitable criterion, such as others users having an establishedrelationship with the user (e.g., based on social data from a socialnetworking service) or others users that have a similar location profileas the user (e.g., family members, work colleagues, friends, etc.).

In this example, the recommendation system can identify one or morecommon actions within the group of users. The recommendation system canthen select one or more of the actions that are to be executed for thegroup of users. In a more particular example, the recommendation systemcan select one or more common actions that are associated with apredetermined number of users (e.g., a majority of the users, a certainpercentage of users in the group, etc.) and select one or more commonactions that may affect the aggregated emotional state of the group ofusers. In another more particular example, the recommendation system canrank the common actions based on any suitable criterion and can thenselect a predetermined number of actions (e.g., top five) and designatethem as group actions. In particular, the common actions can be rankedbased on a deviation between a current profile associated with the userand a target profile so that the recommendation system can determinewhich actions have a higher likelihood of affecting the aggregatedemotional state of the group of users. For example, a high rank can beassigned to a common action that is associated with a greater deviationbetween a current profile and a target profile.

For example, the recommendation system can determine an aggregatedemotional state for a group of users and can then determine that thegroup of users or a threshold number of users within the group arelocated within the proximity of particular output devices. This caninclude determining that the location information associated with theusers in the group of users is within a particular proximity anddetermining the output devices that are connected to or nearby the userdevices associated with each of the co-located users. In a moreparticular example, the group actions can include any suitable actionsthat can be executed, such as presenting suitable media content (e.g., aplaylist of music that may affect the aggregated emotional state of thegroup of users), adjusting ambient light in the surroundings of thegroup of users, adjusting ambient noises and scents in the surroundingsof the group of users, etc.

Upon determining that the recommended action has been executed (e.g.,the device presented a recommended action that included consumingcontent or performing a particular activity), the recommendation systemcan prompt the user of the user device to provide feedback to therecommendation system. For example, the recommendation system canreceive feedback from the user indicating whether the recommended actionwas performed by the user, whether the recommended action may haveimpacted the emotional state of the user, and/or whether the action isto be recommended again to the user. As shown in FIG. 11, therecommendation system can present an interface 1110 that prompts theuser to provide feedback, such as an indication of the change inemotional state, an option to disable an output device, an indication asto whether the user performed a recommended action (e.g., confirmingthat the user walked to a meeting and visited a coffee shop along theway).

Additionally or alternatively, the recommendation system can obtainupdated data, predict the current emotional state of the user and/orgenerate an updated profile, and determine whether the recommendedaction or actions may have moved the user towards one or moreobjectives.

In a more particular implementation, the recommendation system candetermine whether a particular recommended action may have moved theuser towards one or more objectives and/or goals. For example, therecommendation system can prompt the user to provide feedback (e.g.,“How are you feeling now after getting a cup of coffee and watching thatvideo?”) in interface 1110 of FIG. 11. In such an example, therecommendation system can receive feedback from the user relating to theparticular recommended action. In another example, the recommendationsystem can select particular portions of data from multiple data streamsand determine whether the data indicates that the recommended action mayhave moved the user towards one or more objectives. In such an example,in response to receiving authorization from a user of a user device toreceive data relating to the user from multiple data sources, therecommendation system can select data from times subsequent to providingthe recommended action and can determine that the social data andcontextual data indicates the recommended action may have moved the usertowards one or more objectives (e.g., the data indicates that the useris on track to attain a particular activity level).

In some implementations, any suitable rating can be associated with arecommended action. For example, such a rating can include a confidencevalue as to how much the recommendation system believes the recommendedaction may move the user towards one or more objectives. In thisexample, the recommendation system can begin with an initial confidencevalue that is incremented or decremented based on feedback from one ormore users, where the rating can be increased in response to determiningthat a particular user has moved towards an objective after providingthe recommended action. This increased rating can, for example, causethe recommended action to be provided to other users, such as usershaving the same or similar objectives, users placed in the same orsimilar groups as the user, etc. It should be noted that the rating canalso include additional information, such as a difficulty value, atimeliness value, etc.

In accordance with various implementations, mechanisms for presentingcontextually relevant information are provided. In some implementations,the mechanisms for presenting contextually relevant information candetermine that a user is likely to be interested in viewing one or moredocuments based on information related to the user from one or more datasources (e.g., data sources 110). In some implementations, suchdocuments can include a web page, a word processing document, a slideshow presentation, one or more images, a PDF file, an email, etc.

In some implementations, the mechanisms for presenting contextuallyrelevant information can anticipate and identify which documents arelikely to be relevant to the user at any given time. Additionally, insome implementations, the mechanisms for presenting contextuallyrelevant information can cause the identified relevant documents to bepresented to the user such that the user can more easily access theinformation included in the documents.

FIG. 12 shows an example 1200 of a process for presenting contextuallyrelevant information to a user based on information from multiple datasources in accordance with some implementations of the disclosed subjectmatter. As shown in FIG. 12, at 1210, process 1200 can determine one ormore bases on which relevant documents are to be identified and/orpresented based on user input, and when those relevant documents are tobe identified and/or presented based on the user input. For example,process 1200 can determine that documents that are relevant to aparticular task (e.g., a meeting, a project, etc.) are to be identifiedand presented when a user is engaged in that task (and/or is scheduledto be engaged in that task). As another example, process 1200 candetermine that one or more documents that are related to a document thatis currently being presented are to be identified and/or presented asrelevant documents. As yet another example, process 1200 can determinethat documents that are related to a particular task, activity and/orcommon subject matter are to be identified and/or presented as documentsrelevant to that task, activity and/or subject matter. As still anotherexample, process 1200 can determine that one or more documents that wasbeing presented just prior to a currently presented document is to beidentified and/or presented as a relevant document.

At 1220, process 1200 can receive data associated with the user frommultiple data sources. In some implementations, the multiple datasources can include any suitable sources of data, such as sourcesrelated to communications by the user, sources related to scheduleand/or calendar information of the user, and/or any other suitablesources, such as sources described above in connection with 110 of FIG.1 and/or 430 of FIG. 4. In some implementations, a user can be providedwith an opportunity to select which data sources can be used todetermine that one or more documents are contextually relevantdocuments. For example, as described above in connection with FIG. 6, acomputing device associated with the user can present a user interfacethat prompts the user to select and/or de-select data sources that areto be used to identify relevant documents.

At 1230, process 1200 can identify one or more relevant documents basedon the information received from the multiple data sources. In someimplementations, process 1200 can determine which data sources fromamong available data sources to use and can determine portions of theinformation from those data sources that are to be used based on the oneor more bases on which relevant documents are to be identified. Forexample, if process 1200 is to identify documents that are relevant to atask that the user is scheduled to perform, process 1200 can determinethat information is to be used from communications data and scheduledata associated with the user. In some implementations, process 1200 canselect data from only a single data source in order to identify one ormore relevant documents.

In some implementations, process 1200 can use any suitable technique orcombination of techniques to identify one or more relevant documentsbased on the information from the multiple data sources. For example, asdescribed below in connection with FIG. 13, process 1200 (and/or anyother suitable process) can identify relevant documents based oninformation from one or more communications data sources (e.g., emailmessaging, SMS messaging, messaging applications, etc.) that indicatesthat documents were the subject of and/or were attached to one or morecommunications. As another example, as described below in connectionwith FIG. 14, process 1200 (and/or any other suitable process) canidentify relevant documents based on information from one or morecommunications data sources and/or from scheduling information (e.g.,calendar information) that indicates that the documents are associatedwith a particular task and/or activity. As yet another example, asdescribed below in connection with FIG. 15, process 1200 (and/or anyother suitable process) can identify relevant documents based oninformation from a web browser application (and/or any other suitableapplication for presenting documents) that indicates which documents arecurrently available for viewing (e.g., loaded in a tab of a browserwindow and/or in a browser window) and group the available documentsbased on common subject matter of the documents, a common task thatdocuments are associated with, and/or a common activity that documentsare associated with. As still another example, as described below inconnection with FIG. 16, process 1200 (and/or any other suitableprocess) can identify one or more relevant documents based oninformation from a web browser application (and/or any other suitableapplication for presenting documents) that indicates that a new browsertab and/or a new browser window has been opened.

As a further example, process 1200 (and/or any other suitable process)can identify relevant documents based on the author and/or sender of adocument being the same as the author and/or sender of a recently vieweddocument or documents. In such an example, the identified document canbe a document that the user has not yet viewed, a new document, adocument that the user has already reviewed but that also has similarsubject matter (determined using any suitable technique or combinationof techniques), a document having any other suitable properties, or adocument having any suitable combination of these and/or any otherproperties. As another further example, process 1200 (and/or any othersuitable process) can identify relevant documents based on a documentbeing the same type of document as another document or documents that auser has recently viewed. In such an example, the type of document canbe determined based on a subject line of the document and/or of amessage with which the document is associated (e.g., as an attachment toan email), based on the title of the document, based on text or otherdescriptive material in a message to which the document was attached,based on a structure of the document, and/or based on any otherproperties related to the document that can be used to indicate a typefor the document. In a more particular example, if a user has justviewed a purchase order document, process 1200 can identify otherpurchase order documents that may be relevant (e.g., from the samesource, received within a particular period of time, unviewed, notmarked as done, the user has not replied to the message associated withthe document, etc.). As yet another further example, process 1200 canuse any suitable combination of one or more techniques to identifyrelevant documents with the results weighted based on user preferences,user behavior and/or based on any other suitable factor or combinationof factors. In a more particular example, if a user has recently viewedmultiple documents that are of the same type and has responded tomessages associated with those documents, process 1200 can identify oneor more documents at 1230 by identifying other documents of the sametype, where a likelihood that the document is of the same type can beused with a weight associated determines based on a confidence that theuser is likely to be interested in viewing other documents of the sametype (e.g., as opposed to being interested in information related to thesubject of the document and/or sender of the document instead of otherdocuments of the same type). Such a confidence can be determined basedon any suitable information, such as scheduling and/or calendarinformation, messaging information, etc. This weighted informationrelated to document type can be combined with weighted informationgenerated based on process 1200 identifying documents based onscheduling information (e.g., as described below in connection with FIG.14). In such an example, process 1200 can manage presentation of thedocuments (e.g., as described below in connection with 1240) based oncombined weighting information for each document and selecting a subsetof documents (up to and including all documents) that are most relevantbased on the combined weighted information.

At 1240, process 1200 can manage presentation of the one or morerelevant documents identified at 1230 based on information from one ormore of the multiple data sources and/or based on the basis on which thedocument was identified as being relevant. For example, process 1200 cancause the one or more identified relevant documents to be presented inresponse to a user powering on, unlocking and/or otherwise waking up auser device associated with the user within a threshold time of the userparticipating in a conversation (e.g., by email, by SMS, by using amessaging application, etc.) in which the documents were discussedand/or attached. As another example, process 1200 can cause the one ormore identified relevant documents to be presented in response to acurrent time approaching a time at which a task and/or activityassociated with the relevant documents is scheduled to begin. As yetanother example, process 1200 can group together documents that are allrelevant to the same task, activity and/or subject matter for easierviewing of those documents when a user is engaged in the task and/oractivity, and/or interested in viewing documents related to the commonsubject matter. As still another example, process 1200 can present, on asecond display, a document that was being presented on a first displaywhen a new document was opened and presented on the first display.

FIG. 13 shows an example 1300 of a process for presenting documentsrelevant to communications sent and/or viewed using a user deviceassociated with the user (e.g., a user of user device 102) when the userdevice or another user device associated with the same user is poweredon, unlocked, opened, woken from a sleep mode, initiated, etc., and/orwhen a new browser window, and/or a new browser tab are opened inaccordance with some implementations of the disclosed subject matter. At1310, process 1300 can receive information related to communications ofthe user. As described above, for example in connection with 430 of FIG.4, process 1300 can receive the information related to thecommunications of the user subject to one or more restrictions set bythe user. These restrictions can limit which sources of informationrelated to the communication of the user are available for use byprocess 1300 and/or what types of information related to communicationsfrom a particular source are available for use by process 1300. In someimplementations, process 1300 can use information from any suitablecommunication source (e.g., email, SMS messages, messages sent and/orreceived using a messaging application, etc.) and/or can use anysuitable data from those sources (e.g., metadata for messages sentand/or received, identifying information of documents attached to thecommunications, etc.).

At 1320, process 1300 can identify one or more relevant documents basedon the information related to the communications of the user. Process1300 can use any suitable technique or combination of techniques toidentify the one or more relevant documents. For example, process 1300can analyze the information related to communications of the user todetermine whether the user has recently participated in a thread (e.g.,a series of one or more messages exchanged between the user and one ormore other users) associated with one or more documents. In a moreparticular example, process 1300 can use the information receivedrelated to the communications of the user to identify documents thatwere attached to the communications (e.g., by being attached to anemail, by being linked in the body of the communication, etc.). Inanother example, process 1300 can identify all documents associated withthreads in which the user has participated within a threshold period oftime prior to a current time (e.g., within the last minute, two minutes,five minutes, etc.). In yet another example, process 1300 can identifydocuments associated with a thread in which the user participated inwith at least one other user with whom the user is currentlycommunicating (e.g., viewing an email from the other user, composing anemail to the other user, using a messaging application to communicatewith the other user, using a telephone function of the user device toconnect to a contact identified as the other user, etc.). At least aportion of these documents can be identified as relevant documents.

In some implementations, process 1300 can identify multiple relevantdocuments and use any suitable technique or combination or techniques todetermine which amongst the identified relevant documents may berelatively more relevant to a user at a particular time. For example,process 1300 (and/or any other suitable process) can determine which ofthe identified relevant documents are relatively more relevant based ontiming data of when the document (or a version of the document) was lastattached and/or mentioned in a thread. As another example, process 1300(and/or any other suitable process) can determine whether a particulardocument is relatively more relevant based on whether the document (or aversion of the document) was attached and/or mentioned in multiplethreads. As yet another example, process 1300 (and/or any other suitableprocess) can determine whether a particular document is relatively morerelevant based on whether any other documents were attached and/ormentioned after a particular document (or a version of the document) forwhich relevance is being determined was attached or mentioned in athread, and/or how many other documents were attached and/or mentionedin the thread after the particular document was attached and/ormentioned in the thread. As still another example, process 1300 (and/orany other suitable process) can determine whether a particular documentis relatively more relevant by determining whether the document isrelated to other documents that were identified as relevant documents.

At 1330, process 1300 can determine whether a user device associatedwith the user was initiated and/or whether a new browser window and/orbrowser tab has been opened. For example, process 1300 can determinewhether a user device such as a laptop computer is powered on, wokenfrom a sleep mode, opened and/or unlocked. As another example, process1300 can determine whether a user device such as a smartphone or tabletcomputer is powered on, woken from a sleep mode and/or unlocked. As yetanother example, process 1300 can determine whether a browserapplication being executed by a user device has opened a new browserwindow and/or a new browser tab (e.g., within an existing browserwindow). In some implementations, upon determining that a user devicehas been initiated, process 1300 (and/or any other suitable process) canautomatically (e.g., without input from the user to do so) launch abrowser application and/or open a new browser window and/or browser tab.

If a user device associated with the user has not been initiated and ifa new browser window or new browser tab has not been opened (“NO” at1330), process 1300 can return to 1310 and continue to receiveinformation related to communications of the user.

Otherwise, if a user device associated with the user has been initiatedand/or if a new browser window and/or new browser tab has been opened(“YES” at 1330), process 1300 can proceed to 1340.

At 1340, process 1300 can cause the one or more relevant documentsidentified at 1340 to be presented to the user. Process 1300 can use anysuitable technique or combination of techniques to cause the identifieddocuments to be presented. For example, in cases where the user devicehas been initiated, but a browser has not been opened, process 1300 cancause a browser application installed on the user device to be launched.In such an example, process 1300 can cause the browser to navigate to anaddress associated with identified relevant documents. Additionally,process 1300 can cause different documents to be presented in separatetabs. As another example, in cases where a new browser window and/orbrowser tab has been opened by a browser application that is alreadybeing executed, process 1300 can cause the browser to navigate to anaddress associated with any identified documents and present at leastone of the documents using the new browser window and/or tab.

In some implementations, process 1300 can cause a subset of thedocuments identified as relevant documents at 1320 to be presented at1340. For example, as described above in connection with 1320, process1300 can identify which among relevant documents are relatively morerelevant and can cause one or more of the most relevant documents to bepresented.

In some implementations, process 1300 can cause one or more relevantdocuments to be presented as links to the document (e.g., in a home pageor start page of a browser window and/or browser tab), thumbnailsrepresenting the documents, and/or as any other suitable representationof the document rather than presenting a full-sized version of thedocument. Note that although documents are generally described herein asbeing presented by a browser application, such a document can be a webpage that is available on a network such as the internet, but can alsobe a document (e.g., a word processing document, a slideshow document, aPDF document, etc.) other than a web page which can be available locallyand/or remotely.

FIG. 14 shows an example 1400 of a process for presenting documentsrelevant to a scheduled task associated with the user (e.g., a user ofuser device 102) at a time associated with the task in accordance withsome implementations of the disclosed subject matter. At 1410, process1400 can receive information related to communications of the userand/or related to calendar/scheduling information of the user. Asdescribed above, for example in connection with 430 of FIG. 4, process1400 can receive the information related to the communications of theuser and/or calendar/schedule information of the user subject to one ormore restrictions set by the user. These restrictions can limit whichsources of information are available for use by process 1400 and/or whattypes of information are available for use by process 1400. In someimplementations, process 1400 can use information from any suitablecommunication source (e.g., email, SMS messages, messages sent and/orreceived using a messaging application, etc.) and/or calendar/schedulinginformation source, and/or can use any suitable data from those sources(e.g., metadata for messages sent and/or received, identifyinginformation of documents attached to the communications, etc.).

At 1420, process 1400 can identify one or more relevant documents basedat least in part on timing information associated with the informationrelated to communications of the user and/or calendar/schedulinginformation. Process 1400 can use any suitable technique or combinationof techniques to identify the one or more relevant documents. Forexample, process 1400 can analyze the information related tocommunications of the user and the calendar/scheduling information todetermine whether a user is associated with an upcoming event and/ortask such as a meeting, a conference call, an assigned project that theuser is to be working on, etc. In a more particular example, process1400 can use the information received related to the communications ofthe user and calendar/scheduling information to identify documents thatwere attached to communications associated with a particular eventand/or task, and/or documents that are identified in a user'scalendar/schedule information as being relevant to a particular eventand/or task. In another example, process 1400 can identify documentsthat are associated with a particular task and/or event with which theuser has been associated, and process 1400 can determine that thesetasks may be relevant to future occurrences of the same event and/oroccurrences of similar events that are related to that event (e.g., aseries of meetings about the same project, a weekly status meeting,etc.). In yet another example, process 1400 can identify documents thata user has previously opened in association with a similar task and/orevent associated with a similar group of participants (including justthe user), associated with similar subject matter, etc. In someimplementations, at least a portion of the documents identified byprocess 1400 can be identified as relevant documents.

In some implementations, process 1400 can identify multiple relevantdocuments and use any suitable technique or combination of techniques todetermine which amongst the identified relevant documents may berelatively more relevant to a user at a particular time and/or inassociation with a particular task and/or event. For example, process1400 (and/or any other suitable process) can determine which of theidentified relevant documents are relatively more relevant to aparticular task based on how similar the subject matter of the documentis to other documents associated with the task (e.g., other documentsopened by the user, other documents explicitly associated with the taskby the user, etc.). As another example, process 1400 (and/or any othersuitable process) can determine whether a particular document isrelatively more relevant based on whether the document (or a version ofthe document) was attached and/or mentioned in one or more threadsrelated to a particular task and/or event.

At 1430, process 1400 can determine, for documents that are available inmultiple versions (e.g., based on there being revisions to the document,updates to the document, etc.), a version of the one or more relevantdocuments that is appropriate for presentation to the user inassociation with a particular task and/or event. Process 1400 can useany suitable technique or combination of techniques to determine whichof multiple versions of a document is the appropriate version of adocument to present. For example, process 1400 can determine that a mostrecent version of the document is the appropriate document to present tothe user. As another example, process 1400 can determine that a versionof the document that was most recently shared between the user andanother user that is associated with the task and/or activity is theappropriate version of the document to present to the user. As yetanother example, process 1400 can cause one version of the document(e.g., a version that is determined to be most appropriate) to bepresented more prominently, and cause one or more other versions of thedocument to be made available for presentation and/or to be presentedless prominently.

At 1440, process 1400 can determine whether a current time is a propertime to present one or more relevant documents identified at 1420.Process 1400 can determine whether a current time is a proper time usingany suitable technique or combination of techniques. For example,process 1400 can determine whether a current time is within a thresholdtime of a scheduled task and/or event (e.g., five minutes or less from ascheduled start time of a task and/or event, between a scheduled starttime of a task and/or event and a scheduled end time of the task and/orevent, etc.).

If the current time is not a proper time to present one or more relevantdocuments (“NO” at 1440), process 1400 can return to 1410 and continueto receive information associated with the user.

Otherwise, if the current time is a proper time to present one or morerelevant documents (“YES” at 1440), process 1400 can proceed to 1450. At1450, process 1400 can cause one or more relevant documents associatedwith a task and/or event that the user is currently scheduled to perform(and/or that the user will be scheduled to perform in the near future)to be presented. In some implementations, process 1400 can use anysuitable technique or combination of techniques to cause the relevantdocuments to be presented. For example, process 1400 can use techniquesdescribed above in connection with 1240 of FIG. 12 and/or 1340 of FIG.13. As another example, process 1400 can use techniques described belowin connection with FIGS. 15 and/or 16.

FIG. 15 shows an example 1500 of a process for grouping documents basedon information from multiple data sources in accordance with someimplementations of the disclosed subject matter. At 1510, process 1500can receive information associated with the user from multiple datasources. As described above, for example in connection with 430 of FIG.4, process 1500 can receive the information related to any suitable datasource associated with the user subject to one or more restrictions setby the user. These restrictions can limit which sources of informationare available for use by process 1500 and/or what types of informationare available for use by process 1500. In some implementations, process1500 can use information from any suitable data source (e.g., email, SMSmessages, messages sent and/or received using a messaging application, abrowser application, a web search service, a user profile, etc.) and/orcan use any suitable data from those sources (e.g., metadata associatedwith the sources of data, identifying information of documents attachedto communications, data that is representative of the preferences of agroup of users that is similar to the user, etc.).

At 1520, process 1500 can group one or more relevant documents based onthe received information. In some implementations, process 1500 can useany suitable technique or combination of techniques to group the one ormore relevant documents based on the received information. In someimplementations, the relevant documents grouped by process 1500 can bedocuments that are currently open and/or being presented (e.g., by abrowser application, by a word processing application, by one or moreother applications, etc.). Additionally or alternatively, the relevantdocuments grouped by process 1500 can be documents that may or may notbe open and that are identified using any suitable process (e.g., theprocesses described above in connection with FIGS. 13 and 14).

In some implementations, process 1500 can group the one or moredocuments based on information related to communications of the usersuch that documents related to a task, event and/or conversation/threadare grouped.

In some implementations, process 1500 can group one or more documentsbased on metadata of the documents and/or information from a servicethat characterizes documents (e.g., a web search service) indicatingthat the documents are related. For example, multiple documents from thesame web page can be grouped. As another example, multiple documentsthat are related to similar subject matter can be grouped (e.g.,documents that are related to news and/or current events can be grouped,documents that are related to video games can be grouped, documents thatare related to humor can be grouped, etc.). As another example, multipledocuments that have a similar structure can be grouped (e.g., web pagesthat include a single video can be grouped, web pages that include ablog can be grouped, etc.). In some implementations, the same documentcan be placed into multiple groups. For example, a document thatincludes a humorous video can be grouped with other documents thatinclude a video, and can be grouped with other documents that includehumorous subject matter.

At 1530, process 1500 can open one or more browser windows and/orbrowser tabs corresponding to relevant documents and/or relevant groupsof documents and/or can arrange open browser tabs based on the groupingof the relevant documents. For example, process 1500 can arrange browsertabs such that grouped tabs are presented near or adjacent to eachother.

At 1540, process 1500 can cause the browser tabs to be presented basedon the groupings. For example, process 1500 can group one or moredocuments and present a tab corresponding to the group of one or moredocuments. Such a tab can be labeled with a semantically meaningful termthat identifies what the documents associated with the tab are relevantto. In such an example, when a particular tab corresponding to a groupof documents is selected, tabs corresponding to each document can bepresented, and presentation of documents related to other tabscorresponding to groups of documents can be inhibited with only thetab's semantically meaningful label shown.

FIG. 16 shows an example 1600 of a process for presenting a relevantdocument on a second display in accordance with some implementations ofthe disclosed subject matter. As shown in FIG. 16, at 1610, process 1600can receive information associated with the user from multiple datasources. As described above, for example in connection with 430 of FIG.4, process 1600 can receive the information associated with the usersubject to one or more restrictions set by the user. These restrictionscan limit which sources of information are available for use by process1600 and/or what types of information are available for use by process1600. In some implementations, process 1600 can use information from anysuitable source and/or can use any suitable data from those sources(e.g., metadata, identifying information of documents presented by anapplication, devices associated with the user, etc.).

For example, process 1600 can receive information related to a browserapplication of a user. In such an example, process 1600 can receiveinformation indicating that a new browser window and/or a new browsertab has been opened. Process 1600 can also receive informationindicating that one or more documents are being presented by the browserapplication and/or which of these documents (if any) is currently beingpresented to a user.

As another example, process 1600 can receive information indicating thata user device that is being used to present a document on a firstdisplay is capable of causing the document to be presented on a seconddisplay. In such an example, a second display can be a display coupledto the user device, such as a personal computer or a laptop computerthat is coupled to multiple display devices (e.g., having two or moremonitors). Additionally or alternatively, a second display can be adisplay device (such as a television) that the user device can cause topresent content, such as through an application being executed by thedisplay device and/or through a media presentation device (e.g., a mediaplayer, a set-top box, etc.) operatively coupled to the display devicethat the user device can communicate with to cause content to bepresented.

At 1620, process 1600 can determine whether a new browser window and/orbrowser tab has been opened. For example, process 1600 can receiveinformation from the browser application that a new browser windowand/or a new browser tab has been opened. In some implementations,process 1600 can receive information specifying that a new browserwindow and/or a new browser tab has been opened to present a documentthat is related to a document presented just prior to the new browserwindow and/or browser tab being opened. For example, process 1600 canreceive information from the browser application indicating that theuser opened a new tab to load a document that was linked in a currentlypresented browser tab and/or browser window. Additionally, in someimplementations, process 1600 can determine that a new browser taband/or new browser window has been opened only when a document presentedin a new browser window and/or new browser tab is related to a documentpresented just prior to the new browser window and/or browser tab beingopened, and/or satisfies another criteria (e.g., the new tab includes aweb page for a web search service and/or search results from such aservice, the new tab includes information about a term or phrase that auser selected in the currently presented document, etc.)

If process 1600 determines that a new browser window has not been opened(“NO” at 1620), process 1600 can return to 1610 and continue to receiveinformation associated with the user from multiple data sources.Otherwise, if process 1600 determines that a new browser window has beenopened (“YES” at 1620), process 1600 can proceed to 1630.

At 1630, process 1600 can cause a currently presented document to bepresented on a second display based on the information from multipledata sources. For example, process 1600 can determine that a seconddisplay is available to present the currently presented document. Asanother example, process 1600 can determine instructions that are to beused to cause the currently presented document to be presented on thesecond display. In such an example, process 1600 can determine whetherthe user should be queried as to whether to present the currentlypresented document (e.g., to prevent a television that is presenting aprogram that the user is watching from being switched to presenting thecurrently presented document without the user's permission). In someimplementations, when the newly opened browser window and/or browser tabis closed, the document that is presented using the second display canbe inhibited from presentation on the second display, and can instead bepresented on the first display once again.

In some implementations, process 1600 can use any suitable technique orcombination of techniques to cause the currently presented document tobe presented on the second display. For example, in cases where thesecond display is a second monitor coupled to a user device, process1600 can determine that the first display is being used to present thenewly opened browser tab and/or newly opened browser window, and cancause a browser window displaying the currently presented document to bepresented by the second display. As another example, in cases where thesecond display is associated with (and/or is incorporated as part of) amedia presentation device (e.g., a television, a media streaming device,a game console, etc.) that the user device can cause to present content(e.g., through an application being executed by a television, through adevice operatively coupled to an input of a television, etc.), process1600 can determine one or more instructions that are to be sent to causethe media presentation device to present the document that was beingpresented just prior to the new browser window and/or browser tab beingpresented. In such an example, process 1600 can determine that anaddress for the document is to send to the media presentation devicewith an instruction to present the document at the address. In someimplementations, process 1600 can determine that the user device islocated in relatively close proximity to the second display using anysuitable techniques, such as determining that the user device is on thesame local network as the media presentation device, by determining thatthe user device can connect to the media presentation device using shortrange communication techniques (e.g., Bluetooth), by determining that asound emitted by the media presentation device can be sensed by the userdevice, and/or using any other suitable technique or combination oftechniques.

In some implementations, portions of one or more of the processesdescribed above in connection with FIGS. 12-16 can be executed by anysuitable user device. For example, one or more portions of the processesof FIGS. 12-16 can be performed by a server, such as server 120. In suchan example, server 120 can receive the information from the one or moredata sources and use this information to identify relevant documents,and can cause relevant documents to be presented by a user device 102.As another example, one or more portions of processes of FIGS. 12-16 canbe performed by a user device, such as user device 102. In such anexample, user device 102 can receive the information from the one ormore data sources and use this information to identify relevantdocuments, and can present documents and/or cause the documents to bepresented by another user device 102. In some implementations, whichportions are executed by server 120 and which portions are executed byuser device 102 can be determined based on the amount of computationthat is required to execute those portions, based on user preferences,and/or on any other suitable basis or bases.

Accordingly, methods, systems, and media for presenting contextuallyrelevant information are provided.

Although the disclosed subject matter has been described and illustratedin the foregoing illustrative implementations, it is understood that thepresent disclosure has been made only by way of example, and thatnumerous changes in the details of implementation of the disclosedsubject matter can be made without departing from the spirit and scopeof the disclosed subject matter, which is limited only by the claimsthat follow. Features of the disclosed implementations can be combinedand rearranged in various ways.

What is claimed is:
 1. A method for presenting contextually relevantinformation, the method comprising: receiving, using a hardwareprocessor, information associated with a user of a user device from aplurality of data sources; determining, based on at least a portion ofthe received information associated with the user of the user device, ananticipated task and one or more documents that are relevant to theanticipated task; identifying a plurality of relevant documents thatwere (1) previously opened in connection with the anticipated task basedon the received information associated with the user of the user device,and (2) previously opened in connection with tasks that are similar tothe anticipated task, wherein each of the plurality of relevantdocuments is associated with at least one of a plurality of messages ofa messaging account associated with the user of the user device;selecting a subset of relevant documents from the plurality of relevantdocuments for presentation in connection with the anticipated task,wherein the subset of relevant documents is identified based on textincluded in at least one of the plurality of messages of the messagingaccount associated with the user of the user device; and causing a pageto be presented on the user device, wherein the page includes a link toeach of the subset of relevant documents that were previously opened onthe user device.
 2. The method of claim 1, further comprisingdetermining whether a current time is within a particular period of timefrom the time at which the anticipated task is scheduled, wherein thepage is caused to be presented on the user device in response todetermining that the current time is within the particular period oftime from the time at which the anticipated task is scheduled.
 3. Themethod of claim 1, wherein the plurality of data sources includesmetadata of the plurality of messages viewed using the messaging accountassociated with the user.
 4. The method of claim 3, wherein the relevantdocument is a document attached to at least one of the messages, andwherein the relevant document is identified based on the metadataindicating that the relevant document was attached to the at least oneof the messages.
 5. The method of claim 4, wherein the relevant documentis identified based on timing information related to how recently therelevant document was attached to the message.
 6. The method of claim 4,wherein the relevant document is identified based on a number ofmessages in a thread of messages that includes the at least one messageto which the relevant document was attached being equal to or greaterthan a threshold number of messages.
 7. The method of claim 4, whereinattachment to a message is indicated by a link to the relevant documentbeing included in the message.
 8. A system for presenting contextuallyrelevant information, the system comprising: a hardware processor thatis programmed to: receive information associated with a user of a userdevice from a plurality of data sources; determine, based on at least aportion of the received information associated with the user of the userdevice, an anticipated task and one or more documents that are relevantto the anticipated task; identify a plurality of relevant documents thatwere (1) previously opened in connection with the anticipated task basedon the received information associated with the user of the user device,and (2) previously opened in connection with tasks that are similar tothe anticipated task, wherein each of the plurality of relevantdocuments is associated with at least one of a plurality of messages ofa messaging account associated with the user of the user device; selecta subset of relevant documents from the plurality of relevant documentsfor presentation in connection with the anticipated task, wherein thesubset of relevant documents is identified based on text included in atleast one of the plurality of messages of the messaging accountassociated with the user of the user device; and cause a page to bepresented on the user device, wherein the page includes a link to eachof the subset of relevant documents that were previously opened on theuser device.
 9. The system of claim 8, wherein the hardware processor isfurther programmed to determine whether a current time is within aparticular period of time from the time at which the anticipated task isscheduled, wherein the page is caused to be presented on the user devicein response to determining that the current time is within theparticular period of time from the time at which the anticipated task isscheduled.
 10. The system of claim 8, wherein the plurality of datasources includes metadata of the plurality of messages viewed using themessaging account associated with the user.
 11. The system of claim 10,wherein the relevant document is a document attached to at least one ofthe messages, and wherein the relevant document is identified based onthe metadata indicating that the relevant document was attached to theat least one of the messages.
 12. The system of claim 11, wherein therelevant document is identified based on timing information related tohow recently the relevant document was attached to the message.
 13. Thesystem of claim 11, wherein the relevant document is identified based ona number of messages in a thread of messages that includes the at leastone message to which the relevant document was attached being equal toor greater than a threshold number of messages.
 14. The system of claim11, wherein attachment to a message is indicated by a link to therelevant document being included in the message.
 15. A non-transitorycomputer-readable medium containing computer executable instructionsthat, when executed by a processor, cause the processor to perform amethod for presenting contextually relevant information, the methodcomprising: receiving, using a hardware processor, informationassociated with a user of a user device from a plurality of datasources; determining, based on at least a portion of the receivedinformation associated with the user of the user device, an anticipatedtask and one or more documents that are relevant to the anticipatedtask; identifying a plurality of relevant documents that were (1)previously opened in connection with the anticipated task based on thereceived information associated with the user of the user device, and(2) previously opened in connection with tasks that are similar to theanticipated task, wherein each of the plurality of relevant documents isassociated with at least one of a plurality of messages of a messagingaccount associated with the user of the user device; selecting a subsetof relevant documents from the plurality of relevant documents forpresentation in connection with the anticipated task, wherein the subsetof relevant documents is identified based on text included in at leastone of the plurality of messages of the messaging account associatedwith the user of the user device; and causing a page to be presented onthe user device, wherein the page includes a link to each of the subsetof relevant documents that were previously opened on the user device.16. The non-transitory computer-readable medium of claim 15, wherein themethod further comprises determining whether a current time is within aparticular period of time from the time at which the anticipated task isscheduled, wherein the page is caused to be presented on the user devicein response to determining that the current time is within theparticular period of time from the time at which the anticipated task isscheduled.
 17. The non-transitory computer-readable medium of claim 15,wherein the plurality of data sources includes metadata of the pluralityof messages viewed using the messaging account associated with the user.18. The non-transitory computer-readable medium of claim 17, wherein therelevant document is a document attached to at least one of themessages, and wherein the relevant document is identified based on themetadata indicating that the relevant document was attached to the atleast one of the messages.
 19. The non-transitory computer-readablemedium of claim 18, wherein the relevant document is identified based ontiming information related to how recently the relevant document wasattached to the message.
 20. The non-transitory computer-readable mediumof claim 18, wherein the relevant document is identified based on anumber of messages in a thread of messages that includes the at leastone message to which the relevant document was attached being equal toor greater than a threshold number of messages.
 21. The non-transitorycomputer-readable medium of claim 18, wherein attachment to a message isindicated by a link to the relevant document being included in themessage.